dfischermittwald

Qwen3.8-27B-NVFP4-DFlash2

README

License: apache-2.0

Why this exists

vLLM PR #52816 adds DFlash2, a block-diffusion drafter. Its candidate selector runs a TopK over the target model's LM head and refuses a quantized one:

markdown
ValueError: DFlash2 requires an unquantized target LM head for candidate TopK

Every fast NVFP4 Qwen3.8 checkpoint on the Hub quantizes lm_head — unsloth/Qwen3.8-27B-NVFP4 has re:.*lm_head in its FP8 group. So the fastest weights and the best drafter could not be combined at all. This checkpoint is that combination.

Measured

One RTX PRO 6000 Blackwell Server Edition (SM 12.0), TP=1, single stream, min_tokens 400, decode throughput timed separately from TTFT, best of two,

markdown

temperature 1.0 / top_p 0.95 / top_k 20

. Decode tokens/s:

Table with columns: Configuration, size, ctx 2048, ctx 8192, ctx 32768
Configuration	size	ctx 2048	ctx 8192	ctx 32768
this checkpoint + DFlash2 n=7	24 GB	109.7	104.8	110.8
this checkpoint, no drafter	24 GB	51	50	50
`unsloth/Qwen3.8-27B-NVFP4` + MTP n=3	22 GB	83.4	82.9	85.3
`Qwen/Qwen3.8-27B-FP8` + DFlash2 n=7	29 GB	84.9	81.2

Every row is same-node, same-protocol, measured within the same window.

2.15× over autoregressive, and 1.32× over the fastest previously available configuration for this model (unsloth/Qwen3.8-27B-NVFP4 driven by its built-in MTP head), at every context length. The flatness matters as much as the peak: 110.8 tok/s at 32k context, against 49.8 for the FP8 checkpoint with the same drafter, and against MTP on FP8 which at 32k drops to 39.2 — below its own 41.9 autoregressive baseline. DFlash2 keeps paying off where MTP stops.

One number deserves a warning, because it is easy to quote and easy to get wrong. An earlier published measurement of unsloth/Qwen3.8-27B-NVFP4 + MTP n=3 reads 102.3 / 100.4 / 97.0 and was taken with greedy sampling on a different cluster. Greedy raises MTP acceptance (2.2–2.6 there against 2.13 measured here under sampling), so it is not comparable to anything in the table above — DFlash2 cannot do greedy at all, so a greedy row can only ever exist for the MTP arm. Comparing 109.7 against 102.3 understates the difference by more than half; the honest figure is 109.7 against 83.4.

Quantization recipe

The scheme is unsloth/Qwen3.8-27B-NVFP4's, read off their config.json, with exactly one change: re:.*lm_head comes out of the FP8 group and lm_head goes into ignore.

Table with columns: value
	value
`format`	`mixed-precision` (`quant_method: compressed-tensors`)
group_0 — FP8 W8A8	weights 8-bit float, per-channel, static; activations 8-bit float, per-token, dynamic
group_0 targets	`self_attn.(q\|k\|v\|o)_proj`, `linear_attn.(in_proj_qkv\|in_proj_z\|out_proj)`, `layers.(56..63).mlp.(gate\|up\|down)_proj`
group_1 — NVFP4 W4A4	weights 4-bit float, `group_size` 16, `tensor_group`, , float8_e4m3fn; activations 4-bit float, group 16,

Calibration: 512 samples of HuggingFaceH4/ultrachat_200k at 2048 tokens, chat template applied. Tool: llm-compressor.

Two details in unsloth's scheme are what make it 5 GB smaller than a naive port of NVIDIA's NVFP4 recipe, and neither is obvious:

linear_attn is split. The three large projections go to FP8; the two small ones and the norm stay BF16. Qwen3.8-27B is hybrid — about 48 of its 64 layers use linear_attn (Gated DeltaNet) rather than self_attn — so this is where the bytes are.
The last eight layers' MLPs (56–63) are FP8, not NVFP4. Higher precision where it apparently matters more, at a small size cost.

For contrast: NVIDIA's own NVFP4 recipe (from nvidia/Gemma-4-31B-IT-NVFP4) declines to quantize attention at all. Ported faithfully to this model it produces a 29 GB checkpoint — the same size as plain FP8 — and 91.9 tok/s with DFlash2 against this one's 109.7. Gemma has a smaller attention share, so the recipe pays off there and not here.

Size tracks throughput across the whole set, though not as tightly as a single pair of numbers can be made to look: 29 GB against 24 GB is 1.21 on bytes and 51 against 46.1 is 1.11 on autoregressive tokens/s, so size explains most of the gap and not all of it.

The unquantized lm_head costs about 1.3 GB (248320 × 5120, BF16 instead of FP8). That is the whole price of DFlash2 compatibility.

Validation

Correctness. vLLM issue #48898 says NVFP4 kernels can emit garbage or NaN on SM120 without raising, and the engine log confirms this checkpoint selects the named suspect (Using FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM). Six deterministic probes — German factual, arithmetic, fluent German generation, code, plus image shape-counting and OCR against a generated ground-truth image — three runs each, side by side with the known-good unsloth checkpoint: 36/36 passed on both.

Distribution fidelity. Truncated KL divergence against the BF16 original, teacher-forced over five fixed texts, top-20 logprobs per position — see KLD.md in the recipe directory for the numbers and the method's limits. Note it is a truncated KL: vLLM's API exposes at most 20 logprobs, never full logits over the 248320-token vocabulary, so it is a proxy suited to ranking quantizations against each other rather than an absolute figure.

Serving

Needs a vLLM build containing DFlash2. PR #52816 is not in any release as of 2026-08-19; it and its bugfix #52883 are Python and Triton only, so they patch cleanly onto a matching nightly.

bash
vllm serve <path-to-this-checkpoint> \
  --served-model-name Qwen3.8-27B-NVFP4-DFlash2 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --kv-cache-dtype fp8 \
  --max-model-len 65536 \
  --speculative-config '{"method":"dflash","model":"z-lab/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'

Four things that will bite you:

method is dflash, not dflash2. The DFlash2 path is selected by the draft model's architecture (DFlash2DraftModel) in vllm/v1/worker/gpu/spec_decode/__init__.py::init_speculator.
num_speculative_tokens must be 7, i.e. the draft's block_size of 8 minus one.
--async-scheduling is incompatible with method: dflash and raises. vLLM allows it only for EAGLE/MTP/draft_model/NGram/DSpark. This is a genuine structural disadvantage of DFlash2 against MTP in production, not a misconfiguration — and note that all throughput numbers above have it off in every arm, so the comparison is fair.
Greedy sampling is unsupported by the DFlash2 selector. Evaluate at
markdown
```
temperature 1.0 / top_p 0.95 / top_k 20
```
. temperature 0 is an invalid test here, not a stricter one.

Without a DFlash2-capable engine this still serves as an ordinary mixed-precision checkpoint, and its built-in MTP head works too (it is preserved, see below).

The MTP head, and a trap worth knowing about

Qwen3.8-27B ships a built-in multi-token-prediction head as 15 mtp.* tensors inside its regular shards. transformers does not model it — MTP is a vLLM-side module — so from_pretrained never loads them and save_pretrained never writes them. Any llm-compressor round-trip drops the MTP head, and there is no warning anywhere: vLLM starts happily with method=mtp, serves at an acceptance length of exactly 1.00 — every draft token rejected — at 32.3 tok/s, which is below the autoregressive baseline, because it pays for drafting and verification and accepts nothing.

Restoring the tensors is only half the fix. llm-compressor expands ignore patterns against the transformers model, so an re:^mtp.* entry matches zero modules and vanishes from the emitted config; vLLM then builds the head quantized and crashes with no module or parameter named 'fc.weight' in Qwen3_5MultiTokenPredictor. unsloth's checkpoint carries the unexpanded re:^mtp.* in its ignore list for exactly this reason.

This checkpoint has both halves: 15/15 mtp.* tensors present and re:^mtp.* in ignore. Verified serving with method: mtp at acceptance 2.06. If you quantize this model yourself, check for both.

Reproducing

recipes/qwen3-8-27b-nvfp4-dflash2/ in mittwald's internal deploy-quantizations repository, with copy-pasteable re-run instructions in that repo's README. The script gates its own output and fails the job rather than emitting a checkpoint that DFlash2 would refuse or that is missing its MTP head.

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B. Quantization adds no additional restrictions.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

dfischermittwald

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality