Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4 API & Inference Endpoint

Read this first: the missing `mtp.fc.weight`

If you are using any Qwen3.6-27B checkpoint with MTP speculative decoding, check for mtp.fc.weight before you trust a single throughput number.

The upstream base model was re-uploaded on 2026-07-21T18:30:00Z (commit 646f37367f, "Add files using upload-large-folder tool"). That upload added one tensor: mtp.fc.weight, shape [5120, 10240], BF16, 104,857,600 bytes: the fusion projection that concatenates the embedding and hidden streams before the MTP head's single decoder layer. The index at an earlier revision (5ae530c3ab) has 1198 tensors and no mtp.fc.weight; the index at main has 1199 and does. Any checkpoint (upstream or a derivative quantization) pulled before that date has mtp.layers.0.* and mtp.norm but no mtp.fc.

Why this failure is silent. vLLM only enables its strict weight-initialisation check for non-quantized models. From vllm/model_executor/model_loader/default_loader.py:

python
# We only enable strict check for non-quantized models
# that have loaded weights tracking by default.
default_enable_weights_track = (
    model_config.quantization is None and loaded_weights is not None
)

On a quantized checkpoint that check is off, so the missing parameter is never reported. ColumnParallelLinear allocates it, nothing loads into it, and it keeps whatever the allocation left there. The MTP head then drafts from a random projection: no exception, no warning, no error in the log. Acceptance collapses, every drafted token is rejected and re-verified, and you pay the drafting cost for nothing. This card does not contain a timed measurement of that specific broken state, but it does show what very low acceptance costs: the DFlash rows below run at 0.063 acceptance on prose and land 77.0% below the no-speculation baseline at N=64. The only symptom is a number you were not measuring.

vLLM's own source treats this tensor as a known special case (vllm/model_executor/models/qwen3_5_mtp.py):

python
# Workaround: mtp.fc is stored as BF16 in NVFP4 checkpoints but is
# missing from hf_quant_config.json exclude_modules. Force unquantized.
# Ref: https://github.com/vllm-project/vllm/pull/38650
# Ref: https://github.com/NVIDIA/Model-Optimizer/pull/1124

Check your own copy

python
import json
from huggingface_hub import hf_hub_download

repo = "PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4"  # or your own
idx = json.load(open(hf_hub_download(repo, "model.safetensors.index.json")))
wm = idx["weight_map"]

print("tensors:", len(wm))
print("mtp.fc.weight:", "mtp.fc.weight" in wm)
print("mtp.*:", sorted(k for k in wm if k.startswith("mtp.")))

mtp.fc.weight: True is what you need before enabling MTP. If it prints False, do not use --speculative-config '{"method":"mtp",...}' with that checkpoint. You will get a silent slowdown, not an error.

Which build of this model has it

Two artifacts come out of this quantization, and only one of them is safe to run MTP on:

Table with columns: repo, tensors, mtp.fc.weight, MTP
repo	tensors	`mtp.fc.weight`	MTP
`PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4`	2398	absent	do not enable
`PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP`	2399	present	supported

Run the snippet above against whichever one you pulled rather than trusting this table. The MTP throughput rows further down were measured on the -MTP build; the no-speculation and DFlash rows were measured on the plain -NVFP4 build, which is immaterial to those two because neither touches the MTP head.

For a local directory, read model.safetensors.index.json directly. If you want vLLM to fail loudly on any missing tensor rather than trusting the index, force the strict check back on at serve time:

markdown
--model-loader-extra-config '{"enable_weights_track": true}'

(enable_weights_track is an accepted key of --model-loader-extra-config in vLLM 0.25.x; setting it true overrides the quantized-model default and raises ValueError: Following weights were not initialized from checkpoint: {...}.)

Serving

All three modes below were run on the same box with the same flags; only --speculative-config differs.

Common flags used for every measurement in this card:

markdown
--kv-cache-dtype fp8
--attention-backend flashinfer
--max-model-len 262144
--gpu-memory-utilization 0.72
--max-num-seqs 64
--reasoning-parser qwen3
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen3_xml

Every measured run also carried --served-model-name spec-test and a local request-logging middleware (--middleware vllm_streams.StreamTracker) that is not part of vLLM and is not needed to reproduce the serving behaviour; neither is listed above because neither affects how the model is served.

vLLM version under test: 0.25.2.dev0+g752a3a504.d20260721, in the community Grace-Blackwell image ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2.

1. No speculative decoding

bash
vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4 \
  --kv-cache-dtype fp8 --attention-backend flashinfer \
  --max-model-len 262144 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
  --reasoning-parser qwen3 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

The quantization format is read from the checkpoint's own config; no --quantization flag is needed.

2. MTP (multi-token prediction, in-checkpoint draft head)

Add:

markdown
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

This uses the MTP head already inside the checkpoint (text_config.mtp_num_hidden_layers: 1). No second model to download. Requires a checkpoint carrying mtp.fc.weight -- the -MTP build, not the plain -NVFP4 one (see above).

3. DFlash (separate draft model)

Add:

markdown
--speculative-config '{"method":"dflash","model":"<your-dflash-draft>","num_speculative_tokens":15}'

num_speculative_tokens: 15 is z-lab's own recommended value for this draft.

The published DFlash draft for this model family does not load on a released vLLM. z-lab/Qwen3.6-27B-DFlash (1,730,213,120 parameters across 58 tensors, 3,460,432,504 bytes) declares:

json
"layer_types": ["sliding_attention", "sliding_attention",
                "sliding_attention", "sliding_attention", "full_attention"],
"sliding_window": 2048

and _resolve_layer_attention in vllm/model_executor/models/qwen3_dflash.py raises on any mix of sliding and full:

markdown
NotImplementedError: DFlash does not yet support mixed sliding/full attention via
layer_types; see https://github.com/vllm-project/vllm/issues/40898.

The guard reads the draft's layer_types, not the target model's, so a stock unmodified Qwen3.6-27B target hits it identically. This is a vLLM gap, not a property of this checkpoint or of any particular target. z-lab's own card acknowledges it and directs users to install vLLM from the branch of that same PR:

bash
uv pip install -U --torch-backend=auto \
  "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/40898/head"

That is the upstream-sanctioned route, and if you can build from that branch it is the one to use.

Workaround used for the DFlash numbers below, for anyone pinned to a released vLLM: rewrite the draft's config.json so all five entries read "sliding_attention". The weight file is untouched: the draft used here has SHA-256 e0c050b34798d32728a164d2c3f1681746ff85c11945701b0205b654e2f1fdbe, which matches the upstream model.safetensors LFS hash exactly. Only config.json changed.

That mis-declares one layer: it was trained with full attention and is now given a 2048-token window. The target model verifies every drafted token by rejection sampling, so output correctness is unaffected either way; only acceptance can suffer. Note also that a sliding-window layer with a 2048-token window is identical to full attention below 2048 tokens of context, so for short contexts the declaration is exact rather than approximate. Every benchmark in this card ran below that threshold (see Limitations); acceptance beyond 2048 tokens of context is not measured here.

Speculative decoding is not a free speed-up

540 measurement cells: 3 serving configs x 3 workloads x 10 concurrency levels x 6 shuffled repeats, 9,126 streams. All 540 returned status ok, with 0 degraded, 0 failed, 0 errored streams. Run window 2026-07-27T20:49:22Z to 2026-07-28T09:39:38Z on one DGX Spark.

The three configs served the same NVFP4 weights, differing only by the single mtp.fc.weight tensor described above: the MTP rows come from the -MTP build, the base and DFlash rows from the plain -NVFP4 build. Reproducing the MTP rows requires a checkpoint that has the tensor.

Across the 90 (config, workload, concurrency) groups, the coefficient of variation of aggregate tok/s over the 6 repeats has median 0.42%, p90 2.41%, max 5.20%. Differences below a few percent in the tables below are not meaningful; the large ones are.

The three workloads

Chosen to differ in one thing: how predictable the next token is. Each pack generates 64 distinct prompts, so no two concurrent streams at N=64 share a prefix (a repeat would hit prefix caching and flatter the result).

Table with columns: pack, prompt, prompt tokens, intrinsic predictability
pack	prompt	prompt tokens	intrinsic predictability
`prose`	"Write a short story of approximately 1000 words about: <one of 64 topics>. Tell it in the first person..."	57-71	low: every token is a real choice
`code`	"Write a single self-contained Python module implementing <a data structure or algorithm>... full type hints... at least one runnable doctest"	79-94	middle: structured, but generated from scratch
`edit`	"Here is a Python module. Make exactly one change: rename every occurrence of to ... every other line must come back byte-for-byte identical" (followed by the module)

Acceptance is the variable that explains everything

Acceptance rate is the fraction of drafted tokens the target model accepts, taken as a delta of vLLM's spec_decode_num_accepted_tokens_total / spec_decode_num_draft_tokens_total counters around each batch.

Mean acceptance across all ten concurrency levels:

Table with columns: method, prose, code, edit
method	prose	code	edit
MTP	0.475	0.763	1.000
DFlash	0.063	0.187	0.827

(MTP on edit is 0.9998 to four decimal places; it rounds to 1.000 at every level.)

Acceptance is essentially flat in concurrency. It moves with content, not with load:

Table with columns: N, MTP prose, MTP code, MTP edit, DFlash prose, DFlash code, DFlash edit
N	MTP prose	MTP code	MTP edit	DFlash prose	DFlash code	DFlash edit
1	0.492	0.723	1.000	0.066	0.160	0.830
2	0.467	0.783	0.999	0.060	0.199	0.828

This is the point of the whole exercise, and it is the correction worth making to the usual framing: speculative decoding is not "a win at low concurrency that decays at high concurrency". It is a win where acceptance is high, and acceptance is a property of how predictable the output is. Concurrency governs how much spare compute is available to absorb rejected drafts, so it sets the size of the penalty, but the sign of the result is set by acceptance. Rewriting a file you were just given (edit) accepts almost everything; writing original fiction (prose) accepts less than half on MTP and 6% on DFlash. Same box, same flags, same day.

Aggregate throughput

Aggregate output tokens/second, mean of 6 repeats. max_tokens=512, temperature 0.8, thinking disabled; 8,996 of 9,126 streams ran the full budget and 130 stopped early. Token counts come from usage.completion_tokens, not SSE chunk counts (under speculation one chunk can carry several tokens, measured up to 13.1 tokens per chunk on DFlash, so chunk-counting under-reports badly).

prose

Table with columns: N, base, MTP, MTP vs base, DFlash, DFlash vs base
N	base	MTP	MTP vs base	DFlash	DFlash vs base
1	12.6	16.6	+32.3%	16.3	+29.7%
2	25.1	32.8	+30.6%	26.1	+4.0%
4	48.4	61.5

code

Table with columns: N, base, MTP, MTP vs base, DFlash, DFlash vs base
N	base	MTP	MTP vs base	DFlash	DFlash vs base
1	12.6	20.4	+62.4%	27.3	+117.3%
2	25.2	42.9	+70.5%	49.7	+97.3%
4	48.3	80.7

edit

Table with columns: N, base, MTP, MTP vs base, DFlash, DFlash vs base
N	base	MTP	MTP vs base	DFlash	DFlash vs base
1	12.4	24.7	+98.7%	101.5	+716.1%
2	24.7	50.3	+103.9%	160.7	+551.0%
4	46.9	92.6

What the tables say

MTP wins at every concurrency level on code and on edit, including N=64: code +14.9% and edit +34.2% at N=64, with the largest margins at low concurrency (code +62.4%, edit +98.7% at N=1). Acceptance on those workloads is 0.763 and 1.000. There is no crossover to a loss.
MTP is mixed on prose, the workload where its acceptance is lowest (0.475). It wins at N=1,2,4,6,8,24,32 and loses at N=12 (-11.4%), N=16 (-9.5%) and N=64 (-7.2%). At half-acceptance the drafting overhead and the win are close enough that the result flips with batch composition.
DFlash is enormous on edit-shaped work: +716.1% single-stream (101.5 vs 12.4 tok/s) and still +39.4% at N=64, winning at every level, on 0.827 acceptance.
DFlash hits a hard concurrency ceiling of about 21-22 streams on this box, and its flat lines past that are capacity, not speed. Reconstructing from the per-stream timings how many generations actually overlapped, base and MTP ran every level they were asked for (achieved/requested = 1.00 at all ten levels); DFlash reached 16 at N=16 and then stuck at 22, 22, 22 for N=24, 32 and 64. Everything above the ceiling queued. Confirmed live in the engine log: Running: 21 reqs, Waiting: 21 reqs, GPU KV cache usage: 97.2%. So DFlash prose reading 81.3 tok/s at N=64 against base's 353.5 is not a decode rate 77% slower - it is roughly 21 streams' worth of throughput while base runs 64. Judged per running stream the gap is far smaller; judged as a service it is worse than the number looks, because 43 of the 64 requests are waiting rather than generating. Not a scheduler setting: raising from the spec-decode default of 2048 to 7296 moved the ceiling by one stream (22 to 21). It is KV capacity - see the next section.

The KV cost nobody mentions

Speculative decoding takes memory out of the KV pool, which directly limits how much context you can serve. Read from the engine at load time (structural, not timed), at --max-model-len 262144 and --gpu-memory-utilization 0.72 on a 121.7 GiB unified-memory box:

Table with columns: config, weights loaded, KV cache tokens, KV cache, max concurrency at 262144 ctx
config	weights loaded	KV cache tokens	KV cache	max concurrency at 262144 ctx
base	18.76 GiB	2,041,963	63.75 GiB	7.79x
MTP	19.55 GiB	1,839,553	63.02 GiB	7.02x (-10%)
DFlash	22.16 GiB	1,233,978	59.89 GiB	4.71x (-40%)

MTP costs about 10% of the KV pool. DFlash costs about 40%, because it loads a second 3.46 GB model and needs its own KV. If your deployment is context-bound rather than throughput-bound, that is the number to budget against, and it is fixed regardless of workload.

This is also the cause of DFlash's ~21-stream ceiling, and the two costs compound. The pool is 40% smaller, and each running sequence additionally reserves lookahead slots for the 15 draft tokens it proposes per step, so the per-stream KV footprint roughly doubles: dividing the pool by the streams that actually fit gives about 31,900 tokens per stream under base and about 58,800 under DFlash. Fewer tokens shared among hungrier sequences is why the engine reports GPU KV cache usage: 97.2% at 21 concurrent requests and queues the rest. Budget for it directly: on this hardware, at these settings, DFlash serves roughly a third as many simultaneous streams as no-speculation does, whatever the workload. Both figures scale with --gpu-memory-utilization and shrink further at longer --max-model-len.

Per-stream fairness and TTFT at N=64

Aggregate throughput says what the box delivers; it says nothing about what one user waiting on one stream experiences. Per-stream tok/s at N=64 (min / p05 / p50 / p95 / max, mean of 6 repeats):

Table with columns: workload, base, MTP, DFlash
workload	base	MTP	DFlash
prose	5.57 / 5.60 / 5.64 / 5.66 / 5.66	5.25 / 5.28 / 5.44 / 5.67 / 5.84	3.46 / 3.57 / 3.90 / 4.33 / 4.46
code	5.55 / 5.58 / 5.64 / 5.66 / 5.66	6.54 / 6.73 / 6.99 / 7.38 / 7.54	5.08 / 5.60 / 7.34 / 10.50 / 12.28
edit	4.75 / 4.79 / 5.10 / 5.49 / 5.52	6.71 / 6.79 / 7.42 / 8.52 / 8.60	19.62 / 20.20 / 21.77 / 30.72 / 33.78

Base is almost perfectly uniform (max/min 1.02 on prose). Speculation makes streams unequal, because acceptance varies per stream: DFlash on code spans 5.08 to 12.28 tok/s in the same batch.

Time to first token is the clearest symptom of the ceiling, and it is worth reading as a separate cost from decode speed: waiting to start and generating slowly are different failures. Median TTFT at N=64: prose 1.9 s base / 2.6 s MTP / 138.1 s DFlash; code 2.3 / 3.1 / 74.9 s; edit 10.6 / 12.5 / 31.9 s. Base and MTP admit all 64 requests immediately, so their TTFT stays flat; DFlash's median request waits over two minutes for a slot, which no throughput figure shows.

Note this also explains why the DFlash per-stream numbers above look healthier than its aggregate: per-stream rate is measured over each stream's own decode window and therefore excludes queue time entirely. A DFlash stream is fast once it starts. Most of them just start late. If you are serving interactive traffic, size your concurrency to stay under the ceiling rather than relying on the queue.

Provenance and reproduction

Source. nightmedia/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451: bf16, 1199 tensors, 55,457,998,304 bytes declared in its index. Per its own mergekit_config.yml it is a nuslerp merge of Qwen3.6-27B-Architect-Polaris2-Fable-B (weight 1.4) and Qwen3.6-27B-Architect-Polaris-Fable-F451 (weight 0.6), dtype: bfloat16. Uploaded with nightmedia's permission.

Architecture. Qwen3_5ForConditionalGeneration, model_type: qwen3_5. Hybrid attention: 64 decoder layers, 48 linear_attention (gated-delta / Mamba-style) and 16 full_attention, one full-attention layer every fourth (full_attention_interval: 4, at indices 3, 7, ... 63). Hidden size 5120, 24 attention heads, 4 KV heads, head dim 256, intermediate 17408. Vocabulary 248320, max_position_embeddings 262144. Vision tower present (language_model_only: false), so this is a vision-language checkpoint; MTP head is one decoder layer (mtp_num_hidden_layers: 1).

Quantization. NVIDIA TensorRT Model Optimizer 0.45.0 (producer: {"name": "modelopt", "version": "0.45.0"} in hf_quant_config.json), calling examples/llm_ptq/hf_ptq.py directly rather than the huggingface_example.sh wrapper, which never forwards a dataset flag and falls back to a gated default:

bash
python hf_ptq.py \
  --pyt_ckpt_path='<bf16 checkpoint>' \
  --export_path='<out>' \
  --qformat=nvfp4 \
  --calib_size=512 \
  --batch_size=0 \
  --inference_tensor_parallel=1 \
  --dataset cnn_dailymail

Calibration: cnn_dailymail (public), 512 samples. Container: nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev on the DGX Spark itself.

quant_algo: NVFP4, kv_cache_quant_algo: FP8, group_size: 16, weights and input activations both 4-bit float. ModelOpt selected 149 exclude patterns automatically: lm_head, model.language_model.embed_tokens, all 48 linear_attn.conv1d / in_proj_a / in_proj_b blocks, model.visual*, and mtp* / mtp.layers.0*. The linear-attention, vision and MTP paths stay at higher precision; the full-attention and MLP linears are the ones that went to NVFP4.

Sizes. Input 1199 tensors / 55,457,998,304 bytes bf16. ModelOpt export 2398 tensors / 20,487,991,904 bytes -- this is the plain -NVFP4 repo. The -MTP build is that same export plus mtp.fc.weight (104,857,600 bytes, carried through unquantized as BF16 exactly as vLLM expects), giving 2399 tensors / 20,592,849,504 bytes. The two differ by that one tensor and nothing else.

Benchmark harness. specbench.py (matrix runner), workloads.py (the three prompt packs), report.py / dashboard.py (charts), plus the raw 540-cell overnight.json with every per-stream record behind these tables:

TODO before publishing: replace with the harness repository URL. https://github.com/PLACEHOLDER/PLACEHOLDER

The harness is host-side, stdlib-only Python 3.12. It drives docker directly; records usage.completion_tokens rather than SSE chunks; flags any cell with a missing usage block or a generation that stopped short as degraded rather than reporting it as a clean number; and uses at least 64 distinct prompts per pack so no two concurrent streams share a prefix (a repeat would hit prefix caching and flatter the result). Reproducing the exact matrix:

bash
python3 specbench.py --workloads prose,code,edit \
  --levels 1,2,4,6,8,12,16,24,32,64 --repeats 6 --out overnight.json

Credits

nightmedia: the base merge, Qwen3.6-27B-Architect-Polaris2-Fable-B-F451, and its Architect-Polaris merge stages. All the model work is theirs. Uploaded with their permission. That merge in turn credits Qwen/Qwen3.6-27B, DavidAU's Qwen3.5/3.6-27B finetunes, and armand0e's Qwen3.6-27B-Fable-5-Experimental.
Qwen (Alibaba): Qwen3.6-27B, the foundation everything above is built on. Apache-2.0.
z-lab: Qwen3.6-27B-DFlash, the DFlash draft model measured here (MIT licence, per its own repo metadata). The weights used were byte-identical to theirs; only config.json was rewritten to work around a vLLM loader gap.
NVIDIA: TensorRT Model Optimizer 0.45.0, which did the quantization, and the NVFP4 format.

Licence: Apache-2.0, inherited from the base model, which declares license: apache-2.0, as does Qwen/Qwen3.6-27B. The DFlash draft is separately MIT-licensed by z-lab and is not redistributed here.

Limitations

Read these before generalising any number above.

One box, one run. Every measurement comes from a single DGX Spark: GB10, sm_121a, arm64, 121.7 GiB unified memory shared between CPU and GPU. A discrete-GPU system with separate HBM will not reproduce the KV-pool arithmetic, and probably not the crossover points either. There was no second machine and no second run of the full matrix.
One model. These are acceptance rates for this merge, whose MTP head was inherited through a mergekit merge and never retrained against the merged weights. Another Qwen3.6-27B derivative, or the stock Qwen release, will have a different MTP acceptance, possibly much better. Do not read 0.475 on prose as a property of MTP.
Thinking was disabled. Every request passed chat_template_kwargs: {"enable_thinking": false}. Reasoning-mode output has different token statistics and would plausibly have different acceptance.
Short sequences only. Prompts are 57-71 tokens (prose), 79-94 (code) and 820-849 (edit); generation was capped at 512 tokens. The longest sequence in the entire matrix is under 1,400 tokens. Nothing here measures behaviour at 32k, 128k or 262k context, and in particular the DFlash draft's 2048-token sliding window was never exceeded, so the layer_types rewrite described above was exact for these runs and is untested beyond 2048 tokens of context.
Community image, unknown local patches. vLLM 0.25.2.dev0+g752a3a504.d20260721 inside ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2. This is a third-party Grace-Blackwell build, not an official vLLM release; it may carry patches that affect these numbers. A stock vLLM on other hardware may differ.

Read this first: the missing `mtp.fc.weight`

If you are using any Qwen3.6-27B checkpoint with MTP speculative decoding, check for mtp.fc.weight before you trust a single throughput number.

Why this failure is silent. vLLM only enables its strict weight-initialisation check for non-quantized models. From vllm/model_executor/model_loader/default_loader.py:

python
# We only enable strict check for non-quantized models
# that have loaded weights tracking by default.
default_enable_weights_track = (
    model_config.quantization is None and loaded_weights is not None
)

vLLM's own source treats this tensor as a known special case (vllm/model_executor/models/qwen3_5_mtp.py):

python
# Workaround: mtp.fc is stored as BF16 in NVFP4 checkpoints but is
# missing from hf_quant_config.json exclude_modules. Force unquantized.
# Ref: https://github.com/vllm-project/vllm/pull/38650
# Ref: https://github.com/NVIDIA/Model-Optimizer/pull/1124

Check your own copy

python
import json
from huggingface_hub import hf_hub_download

repo = "PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4"  # or your own
idx = json.load(open(hf_hub_download(repo, "model.safetensors.index.json")))
wm = idx["weight_map"]

print("tensors:", len(wm))
print("mtp.fc.weight:", "mtp.fc.weight" in wm)
print("mtp.*:", sorted(k for k in wm if k.startswith("mtp.")))

Which build of this model has it

Two artifacts come out of this quantization, and only one of them is safe to run MTP on:

Table with columns: repo, tensors, mtp.fc.weight, MTP
repo	tensors	`mtp.fc.weight`	MTP
`PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4`	2398	absent	do not enable
`PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4-MTP`	2399	present	supported

markdown
--model-loader-extra-config '{"enable_weights_track": true}'

Serving

All three modes below were run on the same box with the same flags; only --speculative-config differs.

Common flags used for every measurement in this card:

markdown
--kv-cache-dtype fp8
--attention-backend flashinfer
--max-model-len 262144
--gpu-memory-utilization 0.72
--max-num-seqs 64
--reasoning-parser qwen3
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen3_xml

vLLM version under test: 0.25.2.dev0+g752a3a504.d20260721, in the community Grace-Blackwell image ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2.

1. No speculative decoding

bash
vllm serve PassingByPixels/Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4 \
  --kv-cache-dtype fp8 --attention-backend flashinfer \
  --max-model-len 262144 --gpu-memory-utilization 0.72 --max-num-seqs 64 \
  --reasoning-parser qwen3 --trust-remote-code \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml

The quantization format is read from the checkpoint's own config; no --quantization flag is needed.

2. MTP (multi-token prediction, in-checkpoint draft head)

Add:

markdown
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'

3. DFlash (separate draft model)

Add:

markdown
--speculative-config '{"method":"dflash","model":"<your-dflash-draft>","num_speculative_tokens":15}'

num_speculative_tokens: 15 is z-lab's own recommended value for this draft.

The published DFlash draft for this model family does not load on a released vLLM. z-lab/Qwen3.6-27B-DFlash (1,730,213,120 parameters across 58 tensors, 3,460,432,504 bytes) declares:

json
"layer_types": ["sliding_attention", "sliding_attention",
                "sliding_attention", "sliding_attention", "full_attention"],
"sliding_window": 2048

and _resolve_layer_attention in vllm/model_executor/models/qwen3_dflash.py raises on any mix of sliding and full:

markdown
NotImplementedError: DFlash does not yet support mixed sliding/full attention via
layer_types; see https://github.com/vllm-project/vllm/issues/40898.

bash
uv pip install -U --torch-backend=auto \
  "vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/40898/head"

That is the upstream-sanctioned route, and if you can build from that branch it is the one to use.

Speculative decoding is not a free speed-up

The three workloads

Table with columns: pack, prompt, prompt tokens, intrinsic predictability
pack	prompt	prompt tokens	intrinsic predictability
`prose`	"Write a short story of approximately 1000 words about: <one of 64 topics>. Tell it in the first person..."	57-71	low: every token is a real choice
`code`	"Write a single self-contained Python module implementing <a data structure or algorithm>... full type hints... at least one runnable doctest"	79-94	middle: structured, but generated from scratch
`edit`	"Here is a Python module. Make exactly one change: rename every occurrence of to ... every other line must come back byte-for-byte identical" (followed by the module)

Acceptance is the variable that explains everything

Mean acceptance across all ten concurrency levels:

Table with columns: method, prose, code, edit
method	prose	code	edit
MTP	0.475	0.763	1.000
DFlash	0.063	0.187	0.827

(MTP on edit is 0.9998 to four decimal places; it rounds to 1.000 at every level.)

Acceptance is essentially flat in concurrency. It moves with content, not with load:

Table with columns: N, MTP prose, MTP code, MTP edit, DFlash prose, DFlash code, DFlash edit
N	MTP prose	MTP code	MTP edit	DFlash prose	DFlash code	DFlash edit
1	0.492	0.723	1.000	0.066	0.160	0.830
2	0.467	0.783	0.999	0.060	0.199	0.828

Aggregate throughput

prose

Table with columns: N, base, MTP, MTP vs base, DFlash, DFlash vs base
N	base	MTP	MTP vs base	DFlash	DFlash vs base
1	12.6	16.6	+32.3%	16.3	+29.7%
2	25.1	32.8	+30.6%	26.1	+4.0%
4	48.4	61.5

code

Table with columns: N, base, MTP, MTP vs base, DFlash, DFlash vs base
N	base	MTP	MTP vs base	DFlash	DFlash vs base
1	12.6	20.4	+62.4%	27.3	+117.3%
2	25.2	42.9	+70.5%	49.7	+97.3%
4	48.3	80.7

edit

Table with columns: N, base, MTP, MTP vs base, DFlash, DFlash vs base
N	base	MTP	MTP vs base	DFlash	DFlash vs base
1	12.4	24.7	+98.7%	101.5	+716.1%
2	24.7	50.3	+103.9%	160.7	+551.0%
4	46.9	92.6

What the tables say

MTP wins at every concurrency level on code and on edit, including N=64: code +14.9% and edit +34.2% at N=64, with the largest margins at low concurrency (code +62.4%, edit +98.7% at N=1). Acceptance on those workloads is 0.763 and 1.000. There is no crossover to a loss.
MTP is mixed on prose, the workload where its acceptance is lowest (0.475). It wins at N=1,2,4,6,8,24,32 and loses at N=12 (-11.4%), N=16 (-9.5%) and N=64 (-7.2%). At half-acceptance the drafting overhead and the win are close enough that the result flips with batch composition.
DFlash is enormous on edit-shaped work: +716.1% single-stream (101.5 vs 12.4 tok/s) and still +39.4% at N=64, winning at every level, on 0.827 acceptance.
DFlash hits a hard concurrency ceiling of about 21-22 streams on this box, and its flat lines past that are capacity, not speed. Reconstructing from the per-stream timings how many generations actually overlapped, base and MTP ran every level they were asked for (achieved/requested = 1.00 at all ten levels); DFlash reached 16 at N=16 and then stuck at 22, 22, 22 for N=24, 32 and 64. Everything above the ceiling queued. Confirmed live in the engine log: Running: 21 reqs, Waiting: 21 reqs, GPU KV cache usage: 97.2%. So DFlash prose reading 81.3 tok/s at N=64 against base's 353.5 is not a decode rate 77% slower - it is roughly 21 streams' worth of throughput while base runs 64. Judged per running stream the gap is far smaller; judged as a service it is worse than the number looks, because 43 of the 64 requests are waiting rather than generating. Not a scheduler setting: raising from the spec-decode default of 2048 to 7296 moved the ceiling by one stream (22 to 21). It is KV capacity - see the next section.

The KV cost nobody mentions

Table with columns: config, weights loaded, KV cache tokens, KV cache, max concurrency at 262144 ctx
config	weights loaded	KV cache tokens	KV cache	max concurrency at 262144 ctx
base	18.76 GiB	2,041,963	63.75 GiB	7.79x
MTP	19.55 GiB	1,839,553	63.02 GiB	7.02x (-10%)
DFlash	22.16 GiB	1,233,978	59.89 GiB	4.71x (-40%)

Per-stream fairness and TTFT at N=64

Aggregate throughput says what the box delivers; it says nothing about what one user waiting on one stream experiences. Per-stream tok/s at N=64 (min / p05 / p50 / p95 / max, mean of 6 repeats):

Table with columns: workload, base, MTP, DFlash
workload	base	MTP	DFlash
prose	5.57 / 5.60 / 5.64 / 5.66 / 5.66	5.25 / 5.28 / 5.44 / 5.67 / 5.84	3.46 / 3.57 / 3.90 / 4.33 / 4.46
code	5.55 / 5.58 / 5.64 / 5.66 / 5.66	6.54 / 6.73 / 6.99 / 7.38 / 7.54	5.08 / 5.60 / 7.34 / 10.50 / 12.28
edit	4.75 / 4.79 / 5.10 / 5.49 / 5.52	6.71 / 6.79 / 7.42 / 8.52 / 8.60	19.62 / 20.20 / 21.77 / 30.72 / 33.78

Base is almost perfectly uniform (max/min 1.02 on prose). Speculation makes streams unequal, because acceptance varies per stream: DFlash on code spans 5.08 to 12.28 tok/s in the same batch.

Provenance and reproduction

bash
python hf_ptq.py \
  --pyt_ckpt_path='<bf16 checkpoint>' \
  --export_path='<out>' \
  --qformat=nvfp4 \
  --calib_size=512 \
  --batch_size=0 \
  --inference_tensor_parallel=1 \
  --dataset cnn_dailymail

Calibration: cnn_dailymail (public), 512 samples. Container: nvcr.io/nvidia/tensorrt-llm/release:spark-single-gpu-dev on the DGX Spark itself.

TODO before publishing: replace with the harness repository URL. https://github.com/PLACEHOLDER/PLACEHOLDER

bash
python3 specbench.py --workloads prose,code,edit \
  --levels 1,2,4,6,8,12,16,24,32,64 --repeats 6 --out overnight.json

Credits

nightmedia: the base merge, Qwen3.6-27B-Architect-Polaris2-Fable-B-F451, and its Architect-Polaris merge stages. All the model work is theirs. Uploaded with their permission. That merge in turn credits Qwen/Qwen3.6-27B, DavidAU's Qwen3.5/3.6-27B finetunes, and armand0e's Qwen3.6-27B-Fable-5-Experimental.
Qwen (Alibaba): Qwen3.6-27B, the foundation everything above is built on. Apache-2.0.
z-lab: Qwen3.6-27B-DFlash, the DFlash draft model measured here (MIT licence, per its own repo metadata). The weights used were byte-identical to theirs; only config.json was rewritten to work around a vLLM loader gap.
NVIDIA: TensorRT Model Optimizer 0.45.0, which did the quantization, and the NVFP4 format.

Limitations

Read these before generalising any number above.

One box, one run. Every measurement comes from a single DGX Spark: GB10, sm_121a, arm64, 121.7 GiB unified memory shared between CPU and GPU. A discrete-GPU system with separate HBM will not reproduce the KV-pool arithmetic, and probably not the crossover points either. There was no second machine and no second run of the full matrix.
One model. These are acceptance rates for this merge, whose MTP head was inherited through a mergekit merge and never retrained against the merged weights. Another Qwen3.6-27B derivative, or the stock Qwen release, will have a different MTP acceptance, possibly much better. Do not read 0.475 on prose as a property of MTP.
Thinking was disabled. Every request passed chat_template_kwargs: {"enable_thinking": false}. Reasoning-mode output has different token statistics and would plausibly have different acceptance.
Short sequences only. Prompts are 57-71 tokens (prose), 79-94 (code) and 820-849 (edit); generation was capped at 512 tokens. The longest sequence in the entire matrix is under 1,400 tokens. Nothing here measures behaviour at 32k, 128k or 262k context, and in particular the DFlash draft's 2048-token sliding window was never exceeded, so the layer_types rewrite described above was exact for these runs and is untested beyond 2048 tokens of context.
Community image, unknown local patches. vLLM 0.25.2.dev0+g752a3a504.d20260721 inside ghcr.io/timothystewart6/vllm-gb10:v0.25.1-gb10.2. This is a third-party Grace-Blackwell build, not an official vLLM release; it may carry patches that affect these numbers. A stock vLLM on other hardware may differ.

Qwen3.6-27B-Architect-Polaris2-Fable-B-F451-NVFP4

README

Read this first: the missing mtp.fc.weight

Check your own copy

Which build of this model has it

Serving

1. No speculative decoding

2. MTP (multi-token prediction, in-checkpoint draft head)

3. DFlash (separate draft model)

Speculative decoding is not a free speed-up

The three workloads

Acceptance is the variable that explains everything

Aggregate throughput

prose

code

edit

What the tables say

The KV cost nobody mentions

Per-stream fairness and TTFT at N=64

Provenance and reproduction

Credits

Limitations

Explore FriendliAI today

README

Read this first: the missing mtp.fc.weight

Check your own copy

Which build of this model has it

Serving

1. No speculative decoding

2. MTP (multi-token prediction, in-checkpoint draft head)

3. DFlash (separate draft model)

Speculative decoding is not a free speed-up

The three workloads

Acceptance is the variable that explains everything

Aggregate throughput

prose

code

edit

What the tables say

The KV cost nobody mentions

Per-stream fairness and TTFT at N=64

Provenance and reproduction

Credits

Limitations

Read this first: the missing `mtp.fc.weight`

Read this first: the missing `mtp.fc.weight`