bottlecapai

ThinkingCap-Qwen3.6-27B-FP8

README

License: apache-2.0

About this quantization

Native block-wise FP8 (E4M3): weights are quantized to FP8 with 128×128 block scales (weight_scale_inv) and activations are quantized dynamically per token — quant_method: "fp8", weight_block_size: [128, 128], the same DeepSeek-V3-style format that Qwen/Qwen3.6-27B-FP8 ships. vLLM and SGLang load it natively, and the model's MTP (multi-token-prediction) head is preserved, so self-speculative decoding works (≈3.2 accepted tokens/step). ≈29 GB instead of ≈55 GB bf16 — near-lossless quality at half the memory, with FP8 tensor-core throughput on Hopper / Blackwell (Ada / Ampere serve it via weight-only FP8-Marlin kernels).

Kept in bf16: lm_head, the MTP head, the vision tower, and the Gated-DeltaNet input gates (linear_attn.in_proj_a/b, whose 48-wide output doesn't tile a 128 block).

Usage

bash
# vLLM
vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-FP8

# SGLang
python -m sglang.launch_server --model-path bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 --trust-remote-code

Speculative decoding (MTP)

The MTP (multi-token-prediction / NextN) head is kept in bf16, so vLLM can run self-speculative decoding for a decode speed-up — no separate draft model needed. Add --speculative-config when serving (requires vLLM ≥ 0.24.0):

bash
vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-FP8 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Speculative decoding is lossless — the output is identical to standard decoding. It accepts ≈3.35 drafts per verify step here, for a further ≈2.4× on top of the finetune's token savings (see below).

For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF.

Expected performance

Measured on our internal serving harness on 8× RTX PRO 6000 (Blackwell) with vLLM 0.25.0, one configuration per GPU. Each cell is N=200 questions/dataset × 3 generation seeds over a fixed problem subset, batch size 16, sampled decoding (temperature 1.0, top_p 0.95, top_k 20); acc is the mean ± 95% CI across the seeds. Generation budgets are 24,576 tokens (MMLU-Pro) and 16,384 tokens (RealWorldQA).

FP8 matches the bf16 finetune's accuracy (all 95% CIs overlap) while decoding ≈55% faster (36.4 vs 23.4 tok/s, MMLU-Pro standard), and MTP self-speculative decoding (≈3.1–3.3 accepted tokens per verify step) adds a further ≈1.8–2.0× — stacking with the finetune's token savings to ≈6× faster per task than the unquantized base (MMLU-Pro 13.7 s vs 97.3 s; RealWorldQA 6.8 s vs 34.2 s).

All timings are measured per request during the eval. median tokens = median completion length (the finetune's lever); tok/s = per-request steady-state decode rate (prefill excluded via time-to-first-token); task s = measured end-to-end wall-clock per request under the batch-size-16 concurrency; speedup = task s of the base model in standard decoding ÷ task s of the row. Qwen3.6-27B-FP8 (official) is Qwen/Qwen3.6-27B-FP8, the official FP8 quant of the base, shown for comparison.

MMLU-Pro (reasoning)

Table with columns: config, acc, median tokens, tok/s, task s, speedup, accept_len
config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base · standard	0.902 ± 0.019	2186	22.6	97.3	1.00×	—
Qwen3.6-27B base · MTP	0.892 ± 0.031	2117	47.7	45.6	2.13×

RealWorldQA (vision)

Table with columns: config, acc, median tokens, tok/s, task s, speedup, accept_len
config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base · standard	0.802 ± 0.019	718	21.7	34.2	1.00×	—
Qwen3.6-27B base · MTP	0.798 ± 0.064	752	42.2	18.7	1.83×

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

bottlecapai

Model Tree

Base

bottlecapai/ThinkingCap-Qwen3.6-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality