bottlecapai

ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8

README

License: openmdw-1.1

About this quantization

FP8 (E4M3) weights with one scale per output channel, and activations quantized to FP8 per token at run time (compressed-tensors, FP8_DYNAMIC scheme from llm-compressor; recipe in recipe.yaml). The scheme is data-free: no calibration set.

FP8: every linear projection of the 52-layer backbone — the 128 routed experts and the shared expert of each of the 23 mixture-of-experts layers, the input and output projections of the 23 Mamba-2 layers, and the query/key/value/output projections of the 6 attention layers.
bf16: lm_head, the embeddings, the expert router (mixer.gate), the norms, the Mamba-2 convolution and state parameters, and the multi-token-prediction (MTP) block (mtp.*, stored in model-base-aux.safetensors).

In vLLM 0.31.0 the weights take 30.2 GiB of GPU memory.

Usage

Serve it with the base model's recipe, unchanged:

bash

vllm serve bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B-FP8 --async-scheduling \  --reasoning-parser nemotron_v3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

Query it as shown on the main card (OpenAI Chat Completions API; thinking on by default, switched with chat_template_kwargs={"enable_thinking": ...}); use NVIDIA's recommended sampling, temperature=1.0, top_p=0.95.

Tested with vLLM 0.31.0 on an H200. Not yet tested: SGLang, other vLLM versions, and MTP self-speculative decoding (the MTP block is kept in bf16, so it is there to use, but its speed-up on this build is unmeasured).

Known config quirk. quantization_config.ignore in config.json lists the routed experts under transformers' internal module names (model.layers.N.mixer.experts.M), while the checkpoint stores them under backbone.layers.* and in FP8 with their scales. vLLM 0.31.0 ignores those entries and serves the experts with its FP8 W8A8 mixture-of-experts kernels. An engine that honours the list literally could treat the experts as unquantized; that has not been tested.

Expected performance (screen)

What was compared. FP8 against the bf16 model, answering the same questions with the same sampling seeds. Questions are fixed subsets: GPQA-Diamond 60 of 198 questions × 2 seeds, MMLU-Pro 100 of 12,032 × 1, IFBench 40 of 300 × 1, AA-LCR 20 of 100 × 1 (long-document questions; answers graded by Gemma-4-26B-A4B-it with thinking off). Thinking on, NVIDIA's recommended sampling (temperature 1.0, top_p 0.95), 65,536-token generation cap, no speculative decoding. Each model served by vLLM 0.31.0 on one H200 with a bf16 KV cache.

Metrics. Accuracy: the share of question × seed answers scored correct. Δ accuracy: FP8 minus bf16, in percentage points, with a 95% bootstrap interval that resamples questions (seeds averaged per question first). Thinking tokens: length of the reasoning trace, answer excluded, over every question × seed; p95 is its 95th percentile. Δ mean thinking tokens: FP8's mean over bf16's mean on the same questions, minus one, with a question-bootstrap interval.

Table with columns: benchmark (questions × seeds), accuracy %, bf16 → FP8, Δ accuracy, pp [95% CI], thinking tokens mean / median / p95, bf16 → FP8, Δ mean thinking tokens [95% CI]
benchmark (questions × seeds)	accuracy %, bf16 → FP8	Δ accuracy, pp [95% CI]	thinking tokens mean / median / p95, bf16 → FP8	Δ mean thinking tokens [95% CI]
GPQA-Diamond (60 × 2)	77.5 → 75.0	−2.5 [−10.0, +5.0]	11,915 / 4,508 / 42,210 → 11,620 / 4,788 / 37,545	−2.5% [−15.8, +14.2]
MMLU-Pro (100 × 1)	84.0 → 82.0	−2.0 [−7.0, +3.0]	2,936 / 646 / 13,696 → 2,963 / 599 / 20,741	+0.9% [−30.2, +36.6]
IFBench (40 × 1)	40.0 → 45.0	+5.0 [−7.5, +20.0]	8,147 / 4,103 / 19,145 → 6,934 / 2,513 / 23,817	−14.9% [−37.4, +18.7]
AA-LCR (20 × 1)	55.0 → 50.0	−5.0 [−25.0, +10.0]	4,649 / 2,781 / 11,774 → 5,049 / 3,599 / 14,146	+8.6% [−29.3, +59.8]

Does the shorter thinking survive FP8? ThinkingCap against the base model (NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16) quantized with the same recipe, at the same precision, same questions and seeds. Change in mean thinking tokens [95% CI], and the change in accuracy in pp:

Table with columns: benchmark, thinking tokens vs base, bf16, thinking tokens vs base, FP8, Δ accuracy vs base, bf16, Δ accuracy vs base, FP8
benchmark	thinking tokens vs base, bf16	thinking tokens vs base, FP8	Δ accuracy vs base, bf16	Δ accuracy vs base, FP8
GPQA-Diamond	−20.4% [−32.7, −6.8]	−22.9% [−31.0, −13.3]	−0.8 [−5.8, +4.2]	+0.8 [−5.8, +8.3]
MMLU-Pro	−33.6% [−44.7, −16.0]	−36.1% [−52.9, −17.4]	−1.0 [−6.0, +3.0]	−2.0 [−7.0, +3.0]
IFBench	−23.4% [−45.1, +7.8]	−28.4% [−42.4, −9.3]	−7.5 [−22.5, +7.5]	0.0 [−12.5, +12.5]
AA-LCR	−27.0% [−47.4, +10.1]	−10.4% [−38.5, +27.5]	0.0 [−20.0, +20.0]	+5.0 [−15.0, +25.0]

Accuracy: every interval contains zero, so no difference was detected; but the intervals are 10 to 35 pp wide, so the screen cannot rule out a loss of up to 10 pp on GPQA-Diamond or 25 pp on AA-LCR. Evaluated on a screen, not shown lossless.
Thinking length: FP8 does not lengthen the traces detectably on any benchmark. Against the base model at the same precision, the reduction holds on GPQA-Diamond, MMLU-Pro and IFBench (23–36% shorter at FP8, 20–34% at bf16). On AA-LCR it is 10% at FP8 against 27% at bf16, on 20 questions, with both intervals spanning zero.
One trace hit the 65,536-token cap (bf16, IFBench); none for FP8.
The absolute numbers differ from the main card's, which uses the full question sets, more seeds and an FP8 KV cache; compare within these tables only.

Where to find us

Need even more efficiency? Our enterprise versions go further — fewer thinking tokens still, tuned to your workload, at matched accuracy on your own tasks. Deployed on your infrastructure, or in the cloud and region you choose. Talk to our team

License

Released under the base model's own license, the OpenMDW License Agreement 1.1 (full text in LICENSE).

Citation

If you use this model, please cite:

bibtex

@misc{ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B,  title     = {bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B},  author    = {Osusky, Adam and Lindauer, Jan and Jirkovsky, Adam and Mihal, Filip and Platek, Ondrej and Herel, David and Ihnatchenko, Luka and Bartek, Vojtech and Jirak, Jiri and Kubista, Daniel and Krus, Frantisek and Mikolov, Tomas},  year      = {2026},}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

bottlecapai

Model Tree

Base

bottlecapai/ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B

Quantized

this model

Input Modalities

Text

Output Modalities