bottlecapai

ThinkingCap-Qwen3.6-27B-NVFP4

README

License: apache-2.0

About this quantization

Weights are stored at 4 bits in NVIDIA's NVFP4 format (E2M1 values with per-group-of-16 FP8 scales) while activations stay in bf16 — weight-only NVFP4A16 (W4A16), produced data-free with llm-compressor in the compressed-tensors nvfp4-pack-quantized format that vLLM loads natively. ≈19 GB instead of ≈55 GB bf16 — ~2.9× smaller. Being weight-only (W4A16), it serves through vLLM's MarlinNvFp4LinearKernel on both Hopper (H100/H200) and Blackwell: the 4-bit weights unpack to bf16 for compute, so the decode win is memory bandwidth (≈2× bf16 tok/s), not FP4 tensor cores (native cutlass FP4 compute would require a W4A4 / FP4-activation checkpoint). The model's MTP (multi-token-prediction) head is preserved, so self-speculative decoding works (≈3.2 accepted tokens/step).

Kept in bf16: lm_head, the MTP head, the vision tower, and the two small Gated-DeltaNet input gates linear_attn.in_proj_a / in_proj_b (48-wide output, not a multiple of the 4-bit kernel's tile size and fused to width 96 at serve time, so quantizing them breaks loading on non-Blackwell GPUs; keeping them bf16 costs ~60 MB). The remaining group-quantizable Linear layers — MLP, self-attention, and the other linear-attention projections (in_proj_qkv / in_proj_z / out_proj) — carry the 4-bit weights.

Usage

bash
vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4

Speculative decoding (MTP)

The MTP (multi-token-prediction / NextN) head is kept in bf16, so vLLM can run self-speculative decoding for a decode speed-up — no separate draft model needed. Add --speculative-config when serving (requires vLLM ≥ 0.24.0):

bash
vllm serve bottlecapai/ThinkingCap-Qwen3.6-27B-NVFP4 --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Speculative decoding is lossless — the output is identical to standard decoding. It accepts ≈3.2 drafts per verify step here, for a further ≈1.5× on top of the finetune's token savings (see below).

Served on vLLM; SGLang cannot currently load weight-only NVFP4 for this architecture. For local llama.cpp / Ollama / LM Studio use, see the GGUF quantizations at bottlecapai/ThinkingCap-Qwen3.6-27B-GGUF; for near-lossless FP8 at half the memory (loads in vLLM and SGLang), see bottlecapai/ThinkingCap-Qwen3.6-27B-FP8.

Expected performance

Measured on our internal serving harness on 8× RTX PRO 6000 (Blackwell) with vLLM 0.25.0, one configuration per GPU. Each cell is N=200 questions/dataset × 3 generation seeds over a fixed problem subset, batch size 16, sampled decoding (temperature 1.0, top_p 0.95, top_k 20); acc is the mean ± 95% CI across the seeds. Generation budgets are 24,576 tokens (MMLU-Pro) and 16,384 tokens (RealWorldQA).

FP4 matches the bf16 finetune's accuracy (all 95% CIs overlap) while decoding ≈2.3× faster (52.7 vs 22.6 tok/s, MMLU-Pro standard — the memory-bandwidth win of 4-bit weights via the Marlin kernel), and MTP self-speculative decoding (≈3.1–3.2 accepted tokens per verify step) adds a further ≈1.5× — stacking with the finetune's token savings to ≈4.7–8.4× faster per task than the unquantized base (MMLU-Pro 11.6 s vs 97.3 s; RealWorldQA 7.3 s vs 34.2 s).

All timings are measured per request during the eval. median tokens = median completion length (the finetune's lever); tok/s = per-request steady-state decode rate (prefill excluded via time-to-first-token); task s = measured end-to-end wall-clock per request under the batch-size-16 concurrency; speedup = task s of the base model in standard decoding ÷ task s of the row. unsloth NVFP4 and NVIDIA NVFP4 are community NVFP4 quants of the base model, shown for comparison; ThinkingCap-FP8 is our FP8 sibling (bottlecapai/ThinkingCap-Qwen3.6-27B-FP8).

NVFP4 kernel (important for reading tok/s). On this hardware the vLLM NVFP4 kernel is chosen by the checkpoint's FP4 scheme, not a flag, so the NVFP4 rows are not on identical footing: this model and NVIDIA NVFP4 are weight-only W4A16 → MarlinNvFp4LinearKernel (4-bit weights unpacked for compute; a memory-bandwidth decode win), whereas unsloth NVFP4 is W4A4 (weights and activations FP4, mixed-precision) → FlashInferCutlassNvFp4LinearKernel, the cutlass/flashinfer FP4 GEMM. vLLM has no cutlass path for weight-only W4A16 on sm_120, so those fall back to Marlin — a checkpoint-format property, bit-exact-ish and not a source of accuracy difference. On Hopper (H100/H200) there is no native FP4 compute at all, so every NVFP4 serves via Marlin weight-unpack; the cutlass FP4 tensor-core advantage (W4A4) shows mostly on compute-bound prefill / high batch and on datacenter Blackwell.

MMLU-Pro (reasoning)

Table with columns: config, acc, median tokens, tok/s, task s, speedup, accept_len
config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base bf16 · standard	0.902 ± 0.019	2186	22.6	97.3	1.00×	—
Qwen3.6-27B base bf16 · MTP	0.892 ± 0.031	2117	47.7	45.6	2.13×

RealWorldQA (vision)

Table with columns: config, acc, median tokens, tok/s, task s, speedup, accept_len
config	acc	median tokens	tok/s	task s	speedup	accept_len
Qwen3.6-27B base bf16 · standard	0.802 ± 0.019	718	21.7	34.2	1.00×	—
Qwen3.6-27B base bf16 · MTP	0.798 ± 0.064	752	42.2	18.7	1.83×

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

bottlecapai

Model Tree

Base

bottlecapai/ThinkingCap-Qwen3.6-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality