What changed in this upload (e8m8)
Compared with the previous revision (18.2 GB, BF16 head, embedding and MTP block):
lm_head: FP8 W8A8 (per-channel weights, dynamic per-token activations); was BF16.
embed_tokens: INT8 weight-only (symmetric, per-row scale, pack-quantized); was BF16.
- MTP block (
mtp.fc and mtp.layers.* q/k/v/o/gate/up/down projections): FP8 W8A8 per-channel; was BF16.
- Everything else is unchanged: the NVFP4 text layers, vision tower, norms.
- Size: 18.2 GB to 15.30 GB. Shards 1-3 are byte-identical to the previous revision.
Previous revision (original build, runs on stock vLLM 0.33, no patch needed):
806d8306ea57592f4210265577de13afd3813b16
compressed-tensors checkpoint, 15.30 GB, 5 shards. The main group is nvfp4-pack-quantized; the head,
embedding and MTP block have their own FP8 / INT8 groups in config.json.
Table with columns: Part, Precision| Part | Precision |
|---|
| Every linear layer of the 64 text blocks (attention, linear attention, MLP) | NVFP4 weights (E2M1, FP8 E4M3 scale per 16 values, FP32 global scale per tensor) and NVFP4 activations (W4A4) |
| Activation global scales | static, calibrated offline |
Output head (lm_head) | FP8 W8A8: per-channel weights, dynamic per-token activations |
Token embedding (embed_tokens) | INT8 weight-only: symmetric, per-row scale, pack-quantized |
| MTP block linear layers | FP8 W8A8: per-channel weights, dynamic per-token activations |
| Vision tower, linear-attention a/b projections, norms | BF16 (unchanged) |
Fused projections (q/k/v, gate/up, linear-attention qkv/z) share one global scale, so vLLM loads them
without rescaling. The same file also runs as W4A16 (4-bit weights, 16-bit activations) with
vLLM's Marlin kernel, see below.
Use (vLLM 0.33.0)
1. Patch vLLM once (needed for the INT8 embedding)
Stock vLLM 0.33 builds the Qwen3.5 token embedding without a quantization config, so loading this
checkpoint stops at embed_tokens.weight_packed. vllm_patch_embed.py in this repo fixes that: it
passes the quantization config and prefix to the embedding in qwen3_5.py and qwen3_5_mtp.py. It
keeps .orig backups, --undo restores them, and it changes nothing for checkpoints that have no
quantized embedding. Run it with the Python environment that has vLLM installed:
curl -LO https://huggingface.co/aqualaguna/Qwen3.8-23B-Cut-Heal-NVFP4/resolve/main/vllm_patch_embed.pypython vllm_patch_embed.py # python vllm_patch_embed.py --undo to revert
Or skip the patch and use the previous revision (BF16 embedding, 18.2 GB) on stock vLLM, by adding
--revision 806d8306ea57592f4210265577de13afd3813b16 to the command below.
2. Serve
Fast decode (Marlin, 4-bit weights with 16-bit math, MTP drafting; best for chat and agents):
vllm serve aqualaguna/Qwen3.8-23B-Cut-Heal-NVFP4 --linear-backend marlin \ --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \ --kv-cache-dtype fp8 --max-model-len 262144 --gpu-memory-utilization 0.94 \ --reasoning-parser qwen3 --trust-remote-code
Previous revision on stock vLLM (no patch):
vllm serve aqualaguna/Qwen3.8-23B-Cut-Heal-NVFP4 --revision 806d8306ea57592f4210265577de13afd3813b16 \ --linear-backend marlin \ --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \ --kv-cache-dtype fp8 --max-model-len 262144 --gpu-memory-utilization 0.94 \ --reasoning-parser qwen3 --trust-remote-code
Fast prompt reading (native W4A4 on Blackwell's FP4 tensor cores; best for long documents,
batch work): the same command without --linear-backend marlin. The native-kernel numbers below
were measured on the previous revision without drafting and not repeated on this build.
vLLM picks one kernel for the whole run, so you choose: Marlin decodes 35-45% faster, the native
W4A4 kernel reads prompts 1.6-3.5x faster (table below). Chat template and settings as Qwen3.8-27B;
reasoning effort low recommended (see the main card).
Results of this upload (e8m8)
vLLM 0.33, one RTX 5090 32 GB, WSL2, --linear-backend marlin, MTP with 3 draft tokens, fp8 KV cache,
max model length 262,144, GPU memory utilization 0.94, greedy decoding.
Table with columns: Build, File size, GPU load, KV cache tokens, Chat decode tok/s, MTP acceptance| Build | File size | GPU load | KV cache tokens | Chat decode tok/s | MTP acceptance |
|---|
| Previous revision (BF16 head, embedding, MTP block) | 18.2 GB | not measured in this test | | 180.4 | 84.5% |
| FP8 head, BF16 embedding and MTP block (not published; reference for the drift numbers) | 16.93 GB | 16.09 GiB | 321,198 | 224 | 84% |
| e8m8 (this upload) | 15.30 GB | 14.57 GiB | 362,968 | 218 (same as the FP8-head build within noise) | 83% |
(The previous revision without MTP, same card: 16.6 GiB weights and 343,969 KV tokens, see below.)
Other e8m8 numbers with the same setup: short-prompt decode about 185 tok/s, 8k-token prompt about
182 tok/s, 8 parallel streams 1,070 tok/s in total.
Drift against the FP8-head build (mean KL divergence over the top-20 prompt logprobs, 105 texts:
math, code, tool calls, logic, commonsense, spatial reasoning, facts; 13,755 token positions):
Table with columns: Comparison, Mean KL, Top-1 agreement| Comparison | Mean KL | Top-1 agreement |
|---|
| e8m8 vs FP8-head build | 0.0013 | 98.75% |
| Run-to-run noise floor of this test | 0.0011 | 98.95% |
FP8 lm_head alone vs BF16 head | 0.00039 | 99.48% |
The noise floor was measured with a variant that differs only in its MTP draft block, which cannot
change prompt logprobs. The e8m8 drift sits at that floor, so the INT8 embedding adds no measurable
drift. These numbers say how far the output distribution moves between builds; they are not task
accuracy.
Speed on one RTX 5090 (previous revision, no speculative decoding)
This section was measured on the previous revision (BF16 head, embedding and MTP block), before
MTP drafting was used, and is kept for the comparison with the uncut model below.
vLLM 0.33.0 in WSL2, CUDA 13, fp8 KV cache, 262,144 max length, GPU memory utilization 0.94, vision
loaded, no speculative decoding. One request at a time, greedy, 256 output tokens (end of text
ignored), a fresh prompt for every run (no prefix-cache hits), median of 3 runs.
Table with columns: Prompt tokens, Marlin: time to first token, Marlin: decode tok/s, Native W4A4: time to first token, Native W4A4: decode tok/s| Prompt tokens | Marlin: time to first token | Marlin: decode tok/s | Native W4A4: time to first token | Native W4A4: decode tok/s |
|---|
| 29 | 0.04 s | 91.0 | 0.04 s | 67.3 |
| 510 | 0.11 s | 97.2 | 0.06 s | 67.5 |
| 1,975 | 0.38 s | 95.5 | 0.11 s | 67.4 |
| 8,195 | 1.60 s | 95.9 | 0.53 s | 64.6 |
| 32,780 | 7.71 s | 89.1 | 3.07 s | 62.9 |
| 61,780 | 16.71 s | 83.8 | 8.06 s | 61.2 |
| 121,625 | 43.51 s | 78.8 | 27.17 s | 57.2 |
Table with columns: Marlin, Native W4A4 | Marlin | Native W4A4 |
|---|
| Weights in VRAM | 16.6 GiB | 16.6 GiB |
| KV cache | 343,969 tokens | 351,545 tokens |
| 8 requests at once, total output | 691 tok/s (89.7 each) | 499 tok/s (65.5 each) |
Against the uncut Qwen3.8-27B in NVFP4 (published numbers, same card)
Table with columns: This model (previous revision, no drafting), Uncut 27B NVFP4, Source of the uncut number | This model (previous revision, no drafting) | Uncut 27B NVFP4 | Source of the uncut number |
|---|
| Decode, short prompt | 91-97 tok/s (Marlin) | 80.6 | bonnienleohe/Qwen3.8-27B-NVFP4-RTX5090, vLLM 0.27.1, fp8 KV, 256 output tokens |
| Decode, short prompt | | 73.2 | vLLM vs NInfer gist, vLLM 0.27.1 (time includes a 2,048-token prompt) |
| Decode, ~8K context | 95.9 (Marlin) | 81.1 | Infernix README, no drafting |
| Decode, ~122-128K context | 78.8 (Marlin) | 68.4 | Infernix README, no drafting |
| Time to first token, 61,775-token prompt | 8.06 s (native) | 8.52 s | bonnienleohe card |
| Time to first token, 121,625-token prompt | 27.2 s (native) | 26.3 s | bonnienleohe card |
| KV cache, 262,144 max length | 343,969-351,545 tokens (utilization 0.94) | 275,941 (utilization 0.97) | bonnienleohe card |
These are other people's measurements on other machines and engine versions, not a controlled
A/B: read them as "about 15-20% faster decode, about 25% more context, same prompt reading", which
is what a quarter of the FFN weights gone predicts. Decode reads every weight once per token, so
fewer bytes means more tokens per second; the KV gain is the VRAM the smaller weights leave free.
Reading a long prompt is compute-bound and dominated by attention, which the cut did not touch.
One more caveat: the bonnienleohe numbers (80.6 tok/s decode and 8.52 s at 61,775 tokens) come
from one server setting on vLLM 0.27.1. On vLLM 0.33.0 this model gets its faster decode only
with Marlin, which reads prompts about 2x slower; with the native kernel its decode (61-67 tok/s)
is below that published uncut figure. The uncut numbers above are without drafting, so they do not
compare with the 218 tok/s of this build, which uses MTP.
Not available: Infernix and NInfer cannot run this model.
Their kernels are built for the uncut 27B's exact FFN shape (17,408) and refuse 13,312.
Limits
- Accuracy of the NVFP4 version not judged (the main card's results are for the 8-bit GGUF). Only
drift against other builds was measured (above). Expect the usual NVFP4 post-training loss of a
point or two.
- This build needs the vLLM patch above (INT8 embedding); the previous revision does not.
Other engines are untested with the FP8 and INT8 groups.
- The MTP drafter is FP8 and was tested with 3 draft tokens only (83% acceptance).
- One machine, one engine version; numbers move with vLLM releases.
Licence
Apache-2.0, as the base model.