What's in it
Table with columns: tensor, source (BF16), this repo| tensor | source (BF16) | this repo |
|---|
| 400 body linears | BF16 | GPTQ INT4 sym g128 (GPTQModel 7.3.2) |
lm_head (248,320 × 5,120) | BF16, 2.54 GB | GPTQ INT4 sym g128, Hessian-calibrated |
mtp.* draft: fc, q/k/v/o, gate/up/down | BF16 | GPTQ INT4 sym g128, Hessian-calibrated |
embeddings, norms, vision tower, GDN in_proj_a/in_proj_b | BF16 | unchanged (byte-identical) |
17.0 GB on disk. The tokenizer, chat template and processor files are the source repo's, unchanged.
Body. GPTQModel 7.3.2 ran on the Arc Pro B65 itself with these settings:
- 4-bit, symmetric, group 128
desc_act off, act_group_aware on
- damping 0.05, seed 608
- 512 × 2048-token calibration samples: 153 C4, 204 code, 53 chat and 102 tool-call conversations. The tool-call samples are from NousResearch/hermes-function-calling-v1
@dae3e1d, rendered through the model's own chat template with tools=, half of them parallel calls.
The calibration sha256 is in quant-meta.json.
Head and MTP bake. These steps follow the layout of our earlier bakes:
- Forward hooks inside vLLM captured Hessians on wikitext, chat and code slices.
- Each of the nine linears was GPTQ-quantized under a grid of dampings and slice mixes.
- The setting with the lowest output error on held-out rows was kept.
Every baked linear beats round-to-nearest (RTN):
Table with columns: linear, baked, RTN| linear | baked | RTN |
|---|
lm_head | 4.68% | 8.18% |
mtp.fc | 6.98% | 11.47% |
mtp…q_proj | 2.68% | 4.44% |
mtp…k_proj | 6.05% | 10.08% |
mtp…v_proj | 5.11% | 8.28% |
mtp…o_proj | 6.67% | 13.04% |
mtp…gate_proj | 3.67% | 6.35% |
mtp…up_proj | 5.70% | 9.56% |
mtp…down_proj | 7.57% | 11.96% |
Quality vs BF16 (pre-registered)
We wrote the thresholds down and hashed them before any quantized model was evaluated. The BF16 reference ran on a rented
RTX PRO 6000 (vLLM 0.30, CUDA); this checkpoint ran on the Arc Pro B65 (vLLM 0.30, XPU), served exactly as shipped,
with MTP k=6 on for every task and for the logprobs. Full study:
https://launch80.com/a/84829a3a-a0b2-407f-ab38-59cd2d5006fa
Table with columns: #, measure, bar, this checkpoint| # | measure | bar | this checkpoint | |
|---|
| T1 | wikitext-2 perplexity | ≤ +3% | +1.72% | pass |
| T2 | code-corpus perplexity | ≤ +3% | +1.83% | pass |
| T3 | mean next-token KL divergence (1.05M tokens, top-64) | ≤ 0.05 | 0.0412 | pass |
| T4 | top-1 next-token agreement | ≥ 93% | 92.84% | fail (by 0.16 pt) |
| T5 | seven tasks, paired pass@1, greedy | Δ ≥ −2 pts or CI includes 0 | 7/7 tasks (table below) | pass |
| T6 | KLD vs Jackrong's Q4_K_M GGUF | ours ≤ theirs | 0.0412 vs 0.0250 | fail |
| T7 | greedy, spec decoding on = off | identical | 26/32; all 6 differences are at a tie (3 exact, 3 a 0.0156 FP16 step) | fail as written |
Table with columns: task, BF16, this checkpoint, Δ pts [95% CI]| task | BF16 | this checkpoint | Δ pts [95% CI] |
|---|
| GSM8K | 86.96 | 86.66 | −0.30 [−1.21, +0.61] |
| IFEval | 80.22 | 80.78 | +0.55 [−2.22, +3.14] |
| MMLU-Pro subset | 71.80 | 70.90 | −0.90 [−2.70, +0.90] |
| HumanEval+ | 90.24 | 88.41 | −1.83 [−5.49, +1.83] |
| MBPP+ | 76.98 | 75.66 | −1.32 [−3.17, +0.53] |
| BFCL subset (400) | 89.00 | 88.25 | −0.75 [−3.00, +1.50] |
| LiveCodeBench v6 slice | 41.14 | 40.57 | −0.57 [−5.14, +4.00] |
T6, against Jackrong's Q4_K_M GGUF: fail. On the same
512 × 2048 tokens, Q4_K_M stays closer to BF16: mean KLD 0.0250 vs our 0.0412, top-1 agreement 94.92% vs 92.84%, perplexity
+0.73% vs +1.72% (wikitext-2). The files are about the same size (16.8 vs 17.1 GB); Q4_K_M keeps 67 tensors at 6 bits, while
this checkpoint carries a BF16 vision tower and fp16 embeddings and a uniform 4-bit g128 body. Method: the GGUF ran in llama.cpp
(b11723, SYCL) on the B65, its top-64 logprobs were scored with the same code against the same BF16 reference. This differs from
the registered method (a BF16 GGUF base, which needed a rental) and is dated in the thresholds file. Q8_0 measured the same way
gives 0.0024, so the cross-engine term is small. Pick Q4_K_M for fidelity on llama.cpp; pick this build for vLLM with
speculative decoding.
Needle retrieval: 100% at 32k and at 128k. MTP draft acceptance over the whole suite: 46.8% of drafted tokens
(80.2 / 62.2 / 48.0 / 37.4 / 29.5 / 23.3% by draft position), about 3.8 tokens per verify step under greedy decoding.
Disclosure: the tool-call calibration is a post-hoc fix. The first bake, with no tool calls in its calibration set,
passed six of the seven tasks but lost 7.75 points on BFCL. It ended its turn after the first of several parallel calls.
We traced that to the calibration data, recalibrated the body, and rebaked the head and MTP. This checkpoint is that rebake,
and the table above measures it on all seven tasks. The change was made after seeing results, so treat it as post hoc.
The blog post has the whole investigation.
Speed (Arc Pro B65 32 GB, vLLM 0.30.0 XPU, fp8 KV, single stream)
Two serving profiles, same weights. Both are quick BetterBench runs (1 warmup + 5 passes per category):
Table with columns: profile, drafter, context, weighted decode, page| profile | drafter | context | weighted decode | page |
|---|
| fast decode + GDN no-sync patch (v1.1.1) | Launch80/Qwen3.8-27B-DFlash2-Int4-baked-drafter, k=7 | 131,072 | 92.5 / 96.0 tok/s (two runs) | https://launch80.com/a/81dea056-a018-4e48-9f5e-b89b039f368d |
| fast decode, v1.1 (no patch) | same | 131,072 | 89.3 tok/s | https://launch80.com/a/54608011-fd4d-4c68-9d32-868a2fcdd03b |
| full context | built-in MTP head, k=6 | 262,144 | 75.8 tok/s | https://launch80.com/a/30aa8ae2-f00c-409a-a4a9-2e298fc81f67 |
The fast profile gains most on structured output (5.2 accepted tokens per step on JSON, 4.7 on summaries) and least on
long thinking and prose (2.8 and 2.5). It costs context: with the drafter loaded, the 32 GB card fits 128k, not 262k.
Spec-decoding parity (T7) for the fast profile with the v1.1.1 patch: 30 of 32 greedy outputs identical to spec-off, and both
differences are at exact top-2 ties (v1.1 without the patch: 28 of 32, all ties), the same pattern as the MTP profile.
What v1.1.1 fixes: on every speculative step, vLLM's GDN (linear-attention) metadata builder indexed GPU tensors with a CPU
boolean mask. That forces the host to wait for the GPU, which cancelled async scheduling and left ~7 ms of host time in each
step at k=7. patch_gdn_spec_nosync.py replaces the mask with the equivalent slice: 47.9 → 44.4 ms per step on a single
prompt. We also tried a Triton INT4 GEMM for the 9-to-16-row verify shapes; it ran 3.5× slower than oneDNN, so the fast profile
stays at k=7. Quick BetterBench runs vary by a few tok/s between runs, so both runs are shown.
The MTP head is byte-identical to base Qwen3.8-27B (the fine-tune never trained it). We also fine-tuned the DFlash2
drafter on 10,000 of this model's own answers; no checkpoint beat the stock drafter, so v1.1 uses the stock drafter, INT4-baked.
Drafter study: https://launch80.com/a/a7b8a95a-e2c5-473c-b017-310232fc56e0
Serving
Full-context profile (75.8 tok/s, 262k context):
vllm serve Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked \ --quantization gptq --dtype float16 --kv-cache-dtype fp8 --max-model-len 262144 \ --speculative-config '{"method":"mtp","num_speculative_tokens":6}' \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3
Fast-decode profile (92.5 / 96.0 tok/s, 128k context):
vllm serve Launch80/Qwopus3.8-27B-Coder-Flash-Int4-baked \ --quantization gptq --dtype float16 --kv-cache-dtype fp8 --max-model-len 131072 \ --speculative-config '{"method":"dflash","model":"Launch80/Qwen3.8-27B-DFlash2-Int4-baked-drafter","num_speculative_tokens":7,"quantization":"gptq"}' \ --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3
The drafter must run in bf16 while this target runs in fp16: on XPU that needs the patch_dflash_mixed_dtype.py boot
patch from launch80/B65 (on CUDA, serve both in bf16 with --dtype bfloat16).
Every number above was measured with these configurations (vLLM 0.30.0 XPU image), plus three vLLM-XPU correctness
patches for speculative decoding (MTP nightly, MTP boundary, GDN mixed-split) from the same repository. The fast profile's
v1.1.1 number also needs patch_gdn_spec_nosync.py (run it after the MTP boundary patch); without it the same command gives 89.3. Tool calling
needs qwen3_xml, not hermes. Pin a revision (--revision v1.1.1) if you want this exact build; main moves as new
versions ship.
License
Apache-2.0, as the source model and the Qwen3.8-27B base are.
Credits
Qwen (Qwen3.8-27B) · Unsloth (the base Jackrong built on) · Jackrong (Qwopus3.8-27B-Coder-Flash) · NousResearch
(hermes-function-calling-v1, used for calibration) · GPTQModel · Inco AI / z-lab (the DFlash2 drafter) · SpecForge (the drafter fine-tune
experiment) · Wrapzii (asked for this bake, and the on-disk bake
layout it follows).
Questions, results, or a model you want tested? Join the Launch80 Discord: https://discord.gg/launch80
If this work is useful to you, please support the server: https://launch80.com/supporter