Precision map
Table with columns: Component, Precision, Notes| Component | Precision | Notes |
|---|
mlp.{gate,up,down}_proj (64 layers) | NVFP4 W4A16, group 16 | RTN, weight-derived scales |
linear_attn.{in_proj_qkv,in_proj_z,out_proj} (48) | FP8 static W8A8 | calibrated activation scales |
self_attn.{q,k,v,o}_proj (16) | FP8 static W8A8 | calibrated activation scales |
| KV cache | FP8 (E4M3) | calibrated k/v scales |
linear_attn.{in_proj_a,in_proj_b,conv1d,A_log,dt_bias}, norms, embeddings, MTP, vision tower | bf16 | untouched |
lm_head | bf16 | see deviations |
Calibration: 512 samples (~755k tokens, ~30% reasoning traces), seq len 2048.
Calibration data is used only for the FP8-static activation scales and the FP8
KV-cache scales. The NVFP4 weight scales are computed from the weights (RTN) and do
not use calibration — this matches NVIDIA.
Deviations from nvidia/Qwen3.6-27B-NVFP4 (stated explicitly)
lm_head is bf16 here; NVIDIA quantized it to NVFP4. vLLM's
compressed-tensors loader for this architecture does not support a quantized
lm_head (ParallelLMHead exposes only lm_head.weight), so a quantized
lm_head fails to load. This is a forced, format-level deviation — not a tuning
choice. Consequence: this checkpoint is ~0.7 GB larger than NVIDIA's, and keeps
lm_head at higher precision. Any accuracy difference attributable to lm_head is a
memory-budget artifact, not a quality win.- Format: this checkpoint is
compressed-tensors; NVIDIA's is modelopt.
Same underlying NVFP4/FP8 numerics; different loader path in vLLM.
Everything else (which layers are NVFP4 vs FP8 vs bf16, group size 16, FP8-static
activations, FP8 KV cache) matches NVIDIA.
Usage (vLLM)
from vllm import LLM, SamplingParams
llm = LLM(
model="IKilledBardia/Qwen3.6-27B-NVFP4",
trust_remote_code=True,
kv_cache_dtype="fp8",
kernel_config={"enable_flashinfer_autotune": False},
)
print(llm.generate(["<|im_start|>user\nWhat is 84*3/2?<|im_end|>\n<|im_start|>assistant\n"],
SamplingParams(temperature=0, max_tokens=64))[0].outputs[0].text)
Requires a Blackwell GPU (B200 tested). NVFP4 W4A16 runs weight-only via the Marlin
kernel — memory savings, not full FP4-tensor-core compute.
Evaluation
Measured with vLLM. bf16 baseline uses bf16 KV cache; both NVFP4 models use FP8 KV
cache. gsm8k is 0-shot with the chat template and a 4096-token generation budget
(the model reasons in <think> blocks); wikitext perplexity and MMLU are computed
via lm-evaluation-harness (loglikelihood).
Reproduction
Quantized with llm-compressor==0.12.0, compressed-tensors==0.17.0,
transformers==5.10.1; served/evaluated with vllm==0.24.0.