stoyani

Qwen3.8-27B-FP8-KVcal

README

License: apache-2.0

What it fixes

Qwen/Qwen3.8-27B-FP8 ships no KV-cache calibration tensors. With --kv-cache-dtype fp8, vLLM defaults all KV scales to 1.0 and logs kv_cache.py "uncalibrated scaling factor 1.0" warnings — a miscalibrated per-tensor FP8 quantization that assumes K/V activations sit around scale 1.0. On this hybrid (Gated DeltaNet) model the risk is lower than on dense-attention models — K/Q are RMS-normed before hitting the cache, and only 16 of 64 layers are full attention — but the scales are still worth having.

This checkpoint provides, for the 16 full-attention layers (indices 3, 7, 11, ..., 63):

Table with columns: Tensor, Count, Meaning
Tensor	Count	Meaning
`...self_attn.attn.k_scale`	16	calibrated, `amax(K)/448` on observed post-`k_norm` K
`...self_attn.attn.v_scale`	16	calibrated, `amax(V)/448` on observed raw V (no `v_norm` in this architecture)
`...self_attn.attn.q_scale`	16	calibrated, `amax(Q)/448` on observed post-`q_norm` Q
`...self_attn.attn.prob_scale`	16	`float32(1/448)` (P is softmax output, [0,1]; exact scale)

The scales are named along vLLM's runtime internal parameter path (...self_attn.attn.*_scale, not the HF checkpoint convention) so they load unconditionally through the generic weight-name matcher — this works around vllm#54623, where the qwen3_5-family loader never calls maybe_remap_kv_scale_name and silently ignores checkpoint-convention KV scales (fix proposed in vllm#54624). If you maintain a fork/loader, the proper fix is upstream remap support.

Serve it

bash
vllm serve Qwen/Qwen3.8-27B-FP8-KVcal \
  --kv-cache-dtype fp8 \
  --dtype bfloat16 \
  --gpu-memory-utilization 0.94 \
  --enable-chunked-prefill \
  --enable-prefix-caching

Verify at boot: the journal should show all 64 KV scales loaded and no "uncalibrated" scale warnings (kv_cache.py:134/148 "scaling factor 1.0" for K/V, :187 "uncalibrated q_scale / prob_scale"). Verified on the stack this was calibrated for: vLLM 0.23.0, NVIDIA RTX PRO 6000 Blackwell (SM120), CUDA 13.

Expected gains on that hardware (measured, not guaranteed elsewhere): KV pool ~1.95x larger (921K → 1,798K tokens at 0.94 utilization), ~2x residency for long-context concurrent workloads; single-stream tok/s roughly flat (compute-bound regime). FP8 KV does not extend the max context length — the Gated DeltaNet state pool is the memory floor on this hybrid.

Honesty notes

Calibration corpus. Scales were calibrated on 16 documents / 1,100,804 tokens spanning finance, coding, and general text, including two near-full-context documents (262,126 and 262,127 tokens). The corpus is not public (it contains private work documents); the per-layer audit artifacts are in calibration/ (amax + scale per layer, 275 observation calls per layer, status: complete).
Observation point (and why it matters). The scale must match the distribution that is actually quantized into the FP8 cache. In this model K and Q go through per-head RMS norm (k_norm, q_norm, kept BF16) before the cache write; V is a raw projection output — the architecture has no v_norm. This checkpoint's scales are measured at that cache-write point (post-norm K/Q, raw V). Why the distinction matters: e4m3 is a floating format with 4 mantissa bits, so a wrong-but-bounded scale does not produce garbage — it mis-allocates resolution, and the 448-overflow protection only works when scale = amax/448. Independent calibrations that measure pre-norm activations (e.g. scales extracted from unsloth/Qwen3.8-27B-NVFP4, as packaged in joelafrite/Qwen3.8-27B-fp8-KV-scale-calibration) come out a few dozen times larger — pre-norm K sits around 546–596 while post-norm K is 11.6–23.1. That's likely why such numbers still measure acceptably (relative precision forgives a loose scale to a degree), but the post-norm value is the one that is tight at the observed maximum and provably matches what the kernel receives on the verified stack. V's range (14.9–114.6, growing with layer depth) is wider than K's because V is not normed — expected, not a bug.

Benchmark

GSM8K (1,319 test problems, zero-shot, greedy temperature=0, the classic Q: ... A: prompt, no thinking tokens, max_tokens=512), served on NVIDIA RTX PRO 6000 Blackwell (SM120) with vLLM 0.23.0:

Table with columns: KV cache, attention backend, GSM8K accuracy
KV cache	attention backend	GSM8K accuracy
BF16 (control)	FLASH_ATTN	0.630 (831/1319)
FP8, calibrated (this checkpoint)	FLASHINFER	0.643 (848/1319)

Paired result: calibrated FP8 KV shows no degradation versus the BF16 control — net +17 problems in its favor (32 FP8-only correct vs 15 BF16-only), i.e. at-or-above BF16 under this protocol. The headroom of calibrated FP8 is in long-context residency (≈2x KV pool) and long-context decode throughput, not in short-prompt accuracy.

Two reading notes:

Backend differs by hardware constraint, not by choice. On this SM120 card the installed FLASH_ATTN rejects FP8 KV, so the FP8 run is served by FLASHINFER while the BF16 control picks FLASH_ATTN. The accuracy comparison isolates the KV cache dtype (the backends compute the same math); do not read their throughput difference as a KV-dtype effect. On Hopper-class hardware both would use the same backend.
Run-to-run noise. Greedy serving under vLLM's continuous batching is not bit-reproducible across batch configurations (floating-point reduction order); we observed a 3-problem spread between two runs of the exact FP8 set (845 vs 848). Read FP8-vs-BF16 deltas smaller than ~±0.3pp as within run noise.

Reproduce with scripts/eval_gsm8k.py (self-contained: requests + pyarrow, downloads the public parquet from the Hub):

bash
python scripts/eval_gsm8k.py --out-dir gsm8k_results/ \
  --url http://localhost:8000/v1/completions \
  --model Qwen/Qwen3.8-27B-FP8

Rebuild / reproduce

The 64 scales are reproducible with the scripts in scripts/:

calibrate_kv_scales.py — loads the official checkpoint in-process (vLLM V1, enforce_eager), hooks the 16 full-attention layers to record the running |K|/|V| amax that vLLM actually writes to the FP8 cache, runs your calibration corpus (default: 262,144-token context, 4,096-token prefill chunks), and writes kv_cache_scales.safetensors + calibration_report.json. Scale rule: ceil_to_bfloat16(amax / 448), stored float32 (the smallest BF16-representable scale that preserves the observed range). The script refuses input clipping and incomplete observation, and a failed run records status: failed which the injector rejects.
inject_kv_scales.py — copies the official checkpoint (dereferencing LFS symlinks), adds the scale shard, and registers it in model.safetensors.index.json. Requires a complete matching report and 32 finite positive float32 scalars before writing; never modifies weight shards.
add_q_prob_scales.py + — optional follow-up that merges 16 calibrated + 16 into the deployed shard in place, keeping the 32 k/v scales bit-identical (a re-run's k/v output is discarded as calibration noise). File-only; backs up shard + index first and self-checks k/v byte-identity after writing.

Files

75 files: byte-identical to Qwen/Qwen3.8-27B-FP8 (Apache-2.0) — SHA256SUMS lists them all (the official repo's own crc32.txt was found to be internally stale, so we publish a fresh SHA-256 manifest instead)
model.safetensors.index.json: official index + 64 scale entries
kv_cache_scales.safetensors: the 64 calibrated scales (the new content)
calibration/: per-layer audit artifacts (amax, scale, observation counts)
scripts/: the calibration, injection, extend, and eval toolchain
LICENSE: Apache-2.0, matching the upstream weights

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

stoyani

Model Tree

Base

Qwen/Qwen3.8-27B-FP8

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality