About this quantization
Weights are stored as FP8 (E4M3) with per-channel scales; activations are quantized dynamically per token at inference (FP8_DYNAMIC W8A8, produced data-free with llm-compressor) in the compressed-tensors format that vLLM and SGLang load natively. ≈29 GB instead of ≈55 GB bf16 — near-lossless quality at half the memory, with FP8 tensor-core throughput on Ada / Hopper / Blackwell GPUs (older GPUs serve it via weight-only FP8-Marlin kernels).
Kept in bf16: lm_head, MTP heads, and the vision tower.
Usage (vLLM)
vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-FP8
For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF.
From our internal serving-validation harness (vLLM, single-stream, temperature 0) on a fast N=100/dataset subset of MMLU-Pro (reasoning) and RealWorldQA (vision) — a quick quant-parity + decode-speed check, not the headline accuracy evals (for the multi-seed, statistically-tested results see the main model card).
Two things to read off it: FP8 matches the bf16 finetune's accuracy within subset noise while decoding ≈34% faster (85 vs 64 tok/s), and MTP self-speculative decoding (≈3.35 accepted tokens per verify step) adds a further ≈2.4× — stacking with the finetune's token savings to ≈7× faster per task than the unquantized base.
median tokens = median completion length (the finetune's lever); task s = median tokens ÷ single-stream tok/s (real per-request time); speedup is vs the base model in standard decoding. NVIDIA NVFP4 is NVIDIA's ModelOpt NVFP4 quant of the base model, shown for comparison.
MMLU-Pro (reasoning)
Table with columns: config, acc, median tokens, tok/s, task s, speedup, accept_len| config | acc | median tokens | tok/s | task s | speedup | accept_len |
|---|
| Qwen3.6-27B base · standard | 0.86 | 1936 | 62.0 | 31.2 | 1.00× | — |
| Qwen3.6-27B base · MTP | 0.83 | 2042 | 160.6 | 12.7 | 2.46× | 3.32 |
RealWorldQA (vision)
Table with columns: config, acc, median tokens, tok/s, task s, speedup, accept_len| config | acc | median tokens | tok/s | task s | speedup | accept_len |
|---|
| Qwen3.6-27B base · standard | 0.71 | 601 | 62.0 | 9.7 | 1.00× | — |
| Qwen3.6-27B base · MTP | 0.69 | 545 | 160.6 | 3.4 | 2.85× | 3.32 |