stoyani

Qwen3.8-27B-FP8-KVcal

README

License: apache-2.0

What it fixes

Qwen/Qwen3.8-27B-FP8 ships no KV-cache calibration tensors. With --kv-cache-dtype fp8, vLLM defaults all KV scales to 1.0 and logs kv_cache.py "uncalibrated scaling factor 1.0" warnings — a miscalibrated per-tensor FP8 quantization that assumes K/V activations sit around scale 1.0. On this hybrid (Gated DeltaNet) model the risk is lower than on dense-attention models — K/Q are RMS-normed before hitting the cache, and only 16 of 64 layers are full attention — but the scales are still worth having.

This checkpoint provides, for the 16 full-attention layers (indices 3, 7, 11, ..., 63):

Table with columns: Tensor, Count, Meaning
Tensor	Count	Meaning
`...self_attn.attn.k_scale`	16	calibrated, `amax(K)/448` on observed post-`k_norm` K
`...self_attn.attn.v_scale`	16	calibrated, `amax(V)/448` on observed raw V (no `v_norm` in this architecture)
`...self_attn.attn.q_scale`	16	calibrated, `amax(Q)/448` on observed post-`q_norm` Q
`...self_attn.attn.prob_scale`	16	`float32(1/448)` (P is softmax output, [0,1]; exact scale)

The scales are named along vLLM's runtime internal parameter path (...self_attn.attn.*_scale, not the HF checkpoint convention) so they load unconditionally through the generic weight-name matcher — this works around vllm#54623, where the qwen3_5-family loader never calls maybe_remap_kv_scale_name and silently ignores checkpoint-convention KV scales (fix proposed in vllm#54624). If you maintain a fork/loader, the proper fix is upstream remap support.

Serve it

bash

vllm serve stoyani/Qwen3.8-27B-FP8-KVcal \  --kv-cache-dtype fp8 \  --dtype bfloat16 \  --gpu-memory-utilization 0.94 \  --enable-chunked-prefill \  --enable-prefix-caching

Verify at boot: the journal should show all 64 KV scales loaded and no "uncalibrated" scale warnings (kv_cache.py:134/148 "scaling factor 1.0" for K/V, :187 "uncalibrated q_scale / prob_scale"). Verified on the stack this was calibrated for: vLLM 0.23.0, NVIDIA RTX PRO 6000 Blackwell (SM120), CUDA 13.

Expected gains on that hardware (measured, not guaranteed elsewhere): KV pool ~1.95x larger (921K → 1,798K tokens at 0.94 utilization), ~2x residency for long-context concurrent workloads; single-stream tok/s roughly flat (compute-bound regime). FP8 KV does not extend the max context length — the Gated DeltaNet state pool is the memory floor on this hybrid.

Honesty notes

Calibration corpus. Scales were calibrated on 16 documents / 1,100,804 tokens spanning finance, coding, and general text, including two near-full-context documents (262,126 and 262,127 tokens). The corpus is not public (it contains private work documents); the per-layer audit artifacts are in calibration/ (amax + scale per layer, 275 observation calls per layer, status: complete).
Observation point (and why it matters). The scale must match the distribution that is actually quantized into the FP8 cache. In this model K and Q go through per-head RMS norm (k_norm, q_norm, kept BF16) before the cache write; V is a raw projection output — the architecture has no v_norm. This checkpoint's scales are measured at that cache-write point (post-norm K/Q, raw V). Why the distinction matters: e4m3 is a floating format with 4 mantissa bits, so a wrong-but-bounded scale does not produce garbage — it mis-allocates resolution, and the 448-overflow protection only works when scale = amax/448. Independent calibrations that measure pre-norm activations (e.g. scales extracted from unsloth/Qwen3.8-27B-NVFP4, as packaged in joelafrite/Qwen3.8-27B-fp8-KV-scale-calibration) come out a few dozen times larger — pre-norm K sits around 546–596 while post-norm K is 11.6–23.1. That's likely why such numbers still measure acceptably (relative precision forgives a loose scale to a degree), but the post-norm value is the one that is tight at the observed maximum and provably matches what the kernel receives on the verified stack. V's range (14.9–114.6, growing with layer depth) is wider than K's because V is not normed — expected, not a bug.

Benchmark

GSM8K (1,319 test problems, zero-shot, greedy temperature=0, the classic Q: ... A: prompt, no thinking tokens, max_tokens=512), served on NVIDIA RTX PRO 6000 Blackwell (SM120) with vLLM 0.23.0:

Table with columns: KV cache, attention backend, GSM8K accuracy
KV cache	attention backend	GSM8K accuracy
BF16 (control)	FLASH_ATTN	0.630 (831/1319)
FP8, uncalibrated (scale 1.0)	FLASHINFER	0.635 (838/1319)
FP8, calibrated (this checkpoint)	FLASHINFER	0.643 (848/1319)

Paired results: the calibrated checkpoint is at-or-above BF16 (net +17, 32 FP8-only vs 15 BF16-only correct), and uncalibrated FP8 is within run noise of BF16 (net +7). GSM8K is a short-context benchmark — K/V amax on this model sits well under e4m3's 448 ceiling, so scale 1.0 does not overflow, it just under-uses the format's dynamic range, and e4m3's floating precision forgives that at short context.

Two reading notes:

Backend differs by hardware constraint, not by choice. On this SM120 card the installed FLASH_ATTN rejects FP8 KV, so the FP8 run is served by FLASHINFER while the BF16 control picks FLASH_ATTN. The accuracy comparison isolates the KV cache dtype (the backends compute the same math); do not read their throughput difference as a KV-dtype effect. On Hopper-class hardware both would use the same backend.
Run-to-run noise. Greedy serving under vLLM's continuous batching is not bit-reproducible across batch configurations (floating-point reduction order); we observed a 3-problem spread between two runs of the exact FP8 set (845 vs 848). Read FP8-vs-BF16 deltas smaller than ~±0.3pp as within run noise.

RULER long-context (128K)

RULER (NVIDIA) — synthetic long-context benchmark. Prompts generated with the official NVIDIA/RULER generator using this model's own tokenizer (seed 42), n=50 per task at 128K, greedy, max_tokens=128, concurrency 4 (identical prompts and settings for all three configs):

Table with columns: KV config, niah_single_1 (retrieval), vt (variable tracking, 5 chained needles), avg
KV config	niah_single_1 (retrieval)	vt (variable tracking, 5 chained needles)	avg
BF16 (control)	100% (50/50)	61.2% (30/50)	80.6
FP8 uncalibrated	100% (50/50)	62.0% (31/50)	81.0
FP8 calibrated (this checkpoint)	100% (50/50)	66.0 (33/50)	83.0

Reading:

Single-needle retrieval at 128K is flawless under all three KV configs — FP8 KV does not degrade long-range retrieval on this model.
Variable tracking is model-bound, not KV-bound: the BF16 control itself scores only 61% (20/50 samples all configs solve, 8/50 none do). The pairwise deltas (calibrated 33 vs uncalibrated 31 vs BF16 30) favor the calibrated config in every measurement on this and the GSM8K set, but are inside the n=50 noise band — treat as directionally consistent, not a significant difference.
Why uncalibrated is only mildly worse and not broken: e4m3 is a floating format — relative precision is preserved at any magnitude below the 448 ceiling, so a scale-1.0 miscalibration does not corrupt the cache as long as activations stay bounded. On this model they do (post-norm K/V amax ≤ ~115 observed). The calibrated scale's role is overflow headroom, tight range usage, and correct behavior if your workload's activations exceed the calibration range — the failure mode is saturation at 448, not noise.

Net claim: no measurable long-context degradation from FP8 KV on this stack; the calibrated checkpoint is at-or-above the BF16 control in every measurement, and its practical value is the ≈2× KV pool (residency and long-context decode), not an accuracy gain.

Reproduce:

bash

# 1. generate prompts (official RULER generator, model's tokenizer)python /path/to/RULER/scripts/data/prepare.py --save_dir ./ruler \  --benchmark synthetic --task niah_single_1 \  --tokenizer_path <this-repo> --tokenizer_type hf \  --max_seq_length 131100 --model_template_type base --num_samples 50 --random_seed 42# (repeat for --task vt)# 2. score against a running serverpython scripts/eval_ruler.py --data-dir ./ruler --out-dir ./ruler_results --concurrency 4

Reproduce: GSM8K

scripts/eval_gsm8k.py is self-contained (requests + pyarrow, downloads the public parquet from the Hub):

bash

python scripts/eval_gsm8k.py --out-dir gsm8k_results/ \  --url http://localhost:8000/v1/completions \  --model stoyani/Qwen3.8-27B-FP8-KVcal

Rebuild / reproduce

The 64 scales are reproducible with the scripts in scripts/:

calibrate_kv_scales.py — loads the official checkpoint in-process (vLLM V1, enforce_eager), hooks the 16 full-attention layers to record the running |K|/|V| amax that vLLM actually writes to the FP8 cache, runs your calibration corpus (default: 262,144-token context, 4,096-token prefill chunks), and writes kv_cache_scales.safetensors + calibration_report.json. Scale rule: ceil_to_bfloat16(amax / 448), stored float32 (the smallest BF16-representable scale that preserves the observed range). The script refuses input clipping and incomplete observation, and a failed run records status: failed which the injector rejects.
inject_kv_scales.py — copies the official checkpoint (dereferencing LFS symlinks), adds the scale shard, and registers it in model.safetensors.index.json. Requires a complete matching report and 32 finite positive float32 scalars before writing; never modifies weight shards.
add_q_prob_scales.py + — optional follow-up that merges 16 calibrated + 16 into the deployed shard in place, keeping the 32 k/v scales bit-identical (a re-run's k/v output is discarded as calibration noise). File-only; backs up shard + index first and self-checks k/v byte-identity after writing.

Files

75 files: byte-identical to Qwen/Qwen3.8-27B-FP8 (Apache-2.0) — SHA256SUMS lists them all (the official repo's own crc32.txt was found to be internally stale, so we publish a fresh SHA-256 manifest instead)
model.safetensors.index.json: official index + 64 scale entries
kv_cache_scales.safetensors: the 64 calibrated scales (the new content)
calibration/: per-layer audit artifacts (amax, scale, observation counts)
scripts/: the calibration, injection, extend, and eval toolchain
LICENSE: Apache-2.0, matching the upstream weights

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

stoyani

Model Tree

Base

Qwen/Qwen3.8-27B-FP8

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What it fixes

This checkpoint provides, for the 16 full-attention layers (indices 3, 7, 11, ..., 63):

Table with columns: Tensor, Count, Meaning
Tensor	Count	Meaning
`...self_attn.attn.k_scale`	16	calibrated, `amax(K)/448` on observed post-`k_norm` K
`...self_attn.attn.v_scale`	16	calibrated, `amax(V)/448` on observed raw V (no `v_norm` in this architecture)
`...self_attn.attn.q_scale`	16	calibrated, `amax(Q)/448` on observed post-`q_norm` Q
`...self_attn.attn.prob_scale`	16	`float32(1/448)` (P is softmax output, [0,1]; exact scale)

Serve it

bash

vllm serve stoyani/Qwen3.8-27B-FP8-KVcal \  --kv-cache-dtype fp8 \  --dtype bfloat16 \  --gpu-memory-utilization 0.94 \  --enable-chunked-prefill \  --enable-prefix-caching

Honesty notes

Calibration corpus. Scales were calibrated on 16 documents / 1,100,804 tokens spanning finance, coding, and general text, including two near-full-context documents (262,126 and 262,127 tokens). The corpus is not public (it contains private work documents); the per-layer audit artifacts are in calibration/ (amax + scale per layer, 275 observation calls per layer, status: complete).
Observation point (and why it matters). The scale must match the distribution that is actually quantized into the FP8 cache. In this model K and Q go through per-head RMS norm (k_norm, q_norm, kept BF16) before the cache write; V is a raw projection output — the architecture has no v_norm. This checkpoint's scales are measured at that cache-write point (post-norm K/Q, raw V). Why the distinction matters: e4m3 is a floating format with 4 mantissa bits, so a wrong-but-bounded scale does not produce garbage — it mis-allocates resolution, and the 448-overflow protection only works when scale = amax/448. Independent calibrations that measure pre-norm activations (e.g. scales extracted from unsloth/Qwen3.8-27B-NVFP4, as packaged in joelafrite/Qwen3.8-27B-fp8-KV-scale-calibration) come out a few dozen times larger — pre-norm K sits around 546–596 while post-norm K is 11.6–23.1. That's likely why such numbers still measure acceptably (relative precision forgives a loose scale to a degree), but the post-norm value is the one that is tight at the observed maximum and provably matches what the kernel receives on the verified stack. V's range (14.9–114.6, growing with layer depth) is wider than K's because V is not normed — expected, not a bug.

Benchmark

GSM8K (1,319 test problems, zero-shot, greedy temperature=0, the classic Q: ... A: prompt, no thinking tokens, max_tokens=512), served on NVIDIA RTX PRO 6000 Blackwell (SM120) with vLLM 0.23.0:

Table with columns: KV cache, attention backend, GSM8K accuracy
KV cache	attention backend	GSM8K accuracy
BF16 (control)	FLASH_ATTN	0.630 (831/1319)
FP8, uncalibrated (scale 1.0)	FLASHINFER	0.635 (838/1319)
FP8, calibrated (this checkpoint)	FLASHINFER	0.643 (848/1319)

Two reading notes:

Backend differs by hardware constraint, not by choice. On this SM120 card the installed FLASH_ATTN rejects FP8 KV, so the FP8 run is served by FLASHINFER while the BF16 control picks FLASH_ATTN. The accuracy comparison isolates the KV cache dtype (the backends compute the same math); do not read their throughput difference as a KV-dtype effect. On Hopper-class hardware both would use the same backend.
Run-to-run noise. Greedy serving under vLLM's continuous batching is not bit-reproducible across batch configurations (floating-point reduction order); we observed a 3-problem spread between two runs of the exact FP8 set (845 vs 848). Read FP8-vs-BF16 deltas smaller than ~±0.3pp as within run noise.

RULER long-context (128K)

Table with columns: KV config, niah_single_1 (retrieval), vt (variable tracking, 5 chained needles), avg
KV config	niah_single_1 (retrieval)	vt (variable tracking, 5 chained needles)	avg
BF16 (control)	100% (50/50)	61.2% (30/50)	80.6
FP8 uncalibrated	100% (50/50)	62.0% (31/50)	81.0
FP8 calibrated (this checkpoint)	100% (50/50)	66.0 (33/50)	83.0

Reading:

Single-needle retrieval at 128K is flawless under all three KV configs — FP8 KV does not degrade long-range retrieval on this model.
Variable tracking is model-bound, not KV-bound: the BF16 control itself scores only 61% (20/50 samples all configs solve, 8/50 none do). The pairwise deltas (calibrated 33 vs uncalibrated 31 vs BF16 30) favor the calibrated config in every measurement on this and the GSM8K set, but are inside the n=50 noise band — treat as directionally consistent, not a significant difference.
Why uncalibrated is only mildly worse and not broken: e4m3 is a floating format — relative precision is preserved at any magnitude below the 448 ceiling, so a scale-1.0 miscalibration does not corrupt the cache as long as activations stay bounded. On this model they do (post-norm K/V amax ≤ ~115 observed). The calibrated scale's role is overflow headroom, tight range usage, and correct behavior if your workload's activations exceed the calibration range — the failure mode is saturation at 448, not noise.

Reproduce:

bash

# 1. generate prompts (official RULER generator, model's tokenizer)python /path/to/RULER/scripts/data/prepare.py --save_dir ./ruler \  --benchmark synthetic --task niah_single_1 \  --tokenizer_path <this-repo> --tokenizer_type hf \  --max_seq_length 131100 --model_template_type base --num_samples 50 --random_seed 42# (repeat for --task vt)# 2. score against a running serverpython scripts/eval_ruler.py --data-dir ./ruler --out-dir ./ruler_results --concurrency 4

Reproduce: GSM8K

scripts/eval_gsm8k.py is self-contained (requests + pyarrow, downloads the public parquet from the Hub):

bash

python scripts/eval_gsm8k.py --out-dir gsm8k_results/ \  --url http://localhost:8000/v1/completions \  --model stoyani/Qwen3.8-27B-FP8-KVcal

Rebuild / reproduce

The 64 scales are reproducible with the scripts in scripts/:

calibrate_kv_scales.py — loads the official checkpoint in-process (vLLM V1, enforce_eager), hooks the 16 full-attention layers to record the running |K|/|V| amax that vLLM actually writes to the FP8 cache, runs your calibration corpus (default: 262,144-token context, 4,096-token prefill chunks), and writes kv_cache_scales.safetensors + calibration_report.json. Scale rule: ceil_to_bfloat16(amax / 448), stored float32 (the smallest BF16-representable scale that preserves the observed range). The script refuses input clipping and incomplete observation, and a failed run records status: failed which the injector rejects.
inject_kv_scales.py — copies the official checkpoint (dereferencing LFS symlinks), adds the scale shard, and registers it in model.safetensors.index.json. Requires a complete matching report and 32 finite positive float32 scalars before writing; never modifies weight shards.
add_q_prob_scales.py + — optional follow-up that merges 16 calibrated + 16 into the deployed shard in place, keeping the 32 k/v scales bit-identical (a re-run's k/v output is discarded as calibration noise). File-only; backs up shard + index first and self-checks k/v byte-identity after writing.

Files

75 files: byte-identical to Qwen/Qwen3.8-27B-FP8 (Apache-2.0) — SHA256SUMS lists them all (the official repo's own crc32.txt was found to be internally stale, so we publish a fresh SHA-256 manifest instead)
model.safetensors.index.json: official index + 64 scale entries
kv_cache_scales.safetensors: the 64 calibrated scales (the new content)
calibration/: per-layer audit artifacts (amax, scale, observation counts)
scripts/: the calibration, injection, extend, and eval toolchain
LICENSE: Apache-2.0, matching the upstream weights