GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
nota-ai
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
nota-ai
Model Tree
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Input Modalities
Output Modalities
Supported Functionality
| NVFP4 | Ours | |
|---|---|---|
| MoE layers | 23 | 23 |
| routed experts per layer | 128 (uniform) | 96 / 100 / 116 (per layer) |
| total routed experts | 2,944 | 2,484 (−15.6%) |
| experts per token (top-k) | 6 | 6 |
| shared experts | 1 | 1 |
| MTP block | 128 experts | 128 experts (unpruned) |
| safetensors on disk | 20.08 GiB | 17.67 GiB |
Capacity is concentrated at the two ends of the stack: the first and last six MoE layers keep 116 experts each, and the eleven middle layers carry the whole reduction at 96–100.
Only MoE experts are pruned. The 6 attention layers, 23 Mamba-2 layers, shared experts and the MTP block are untouched, so the 1M-token context window is unchanged.
Every layer's expert count is a multiple of 4, so tensor/expert parallel sizes of
2 and 4 divide evenly. Counts live in config.json as
n_routed_experts_per_layer; n_routed_experts stays at 128 — the largest count
in the model, held by the unpruned MTP block — so weight mapping still covers
every expert.
Quantization is inherited from the base checkpoint and not modified:
W4A16_NVFP4 with group_size=16 for the experts (4-bit weights, bf16
activations) and FP8 for a small set of Mamba projections, in NVIDIA ModelOpt
format.
Greedy decoding (temperature 0), context length 139,264 tokens, generation
budget max_tokens 131,072, full benchmark sets.
Greedy decoding is used deliberately: it removes sampling variance, so the degradation introduced by compression is measured reproducibly rather than blurred by run-to-run noise.
| benchmark | NVFP4 | REAP (ICLR 2026) | Ours |
|---|---|---|---|
| GPQA-Diamond | 74.24% | 63.64% | 67.68% |
| MMLU-Pro | 80.71% | 76.08% | 76.60% |
| IFEval | 92.05% | 86.88% | 87.25% |
| IFBench | 70.00% | 65.67% | 65.00% |
Per-layer allocation lifts the average by +1.58 points over REAP and leads on 4 of the 5 benchmarks; HumanEval+ ends up above the unpruned NVFP4 checkpoint.
Note. REAP is the state of the art for MoE expert pruning and the method behind most pruned MoE checkpoints published on the Hugging Face Hub. The column above is a REAP baseline built from the same saliency scores and the same 15.6% removal rate as ours, removed uniformly from every layer — so the only difference is how the budget is spread across layers.
Measured with vLLM 0.27.1, enforce_eager, serving the full 1M-token context
window.
| on a 32 GB budget | NVFP4 | Ours | change |
|---|---|---|---|
| weights + non-torch | 19.92 GiB | 17.42 GiB | −2.50 GiB |
| peak activation | 2.26 GiB | 2.26 GiB | — |
| KV cache available | 6.43 GiB | 8.93 GiB | +39% |
| KV cache capacity | 2,199,552 tokens | 3,059,712 tokens |
Both checkpoints start on a single 32 GB card. Pruning does not change whether the model fits — it changes how much of the card is left for serving.
Two things worth knowing:
--kv-cache-dtype fp8 does not enlarge the KV pool here. vLLM pads the
attention page up to the Mamba state page (text
attention block size 4176 tokens to ensure that attention page size is >= mamba page sizeauto and fp8 both give 10.28 GiB /
3,522,560 tokens / 3.36× concurrency on this checkpoint.shell
export MODEL_CKPT=nota-ai/Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15export VLLM_PATH=/path/to/vllmvLLM already calls a get_nemotron_h_config_for_layer hook for models with
per-layer expert counts, but the transformers config class that must answer that
hook does not implement it. Copy one file over the installed one:
shell
cp patch/configuration_nemotron_h.py \ $($VLLM_PATH/bin/python -c "import transformers.models.nemotron_h.configuration_nemotron_h as m; print(m.__file__)")Without it, loading fails with
AssertionError: Attempted to load weight (torch.Size([116])) into parameter (torch.Size([128])),
because every MoE layer would be built with the maximum expert count instead of
its own. Re-apply after any pip install -U transformers.
The deployment recipes from the base model card
carry over unchanged — pruning removes experts but leaves the architecture,
tokenizer, chat template and MTP block intact. vLLM v0.27.1 or newer.
Max throughput, single GPU:
shell
vllm serve --model $MODEL_CKPT \ --max-num-seqs 256 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --async-scheduling \ --mamba-backend flashinfer \ --mamba-cache-mode align \ --reasoning-parser nemotron_v3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choiceInteractive, low concurrency, with the DSpark drafter for speculative
decoding (export DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark):
shell
vllm serve --model $MODEL_CKPT \ --max-num-seqs 128 \ --enable-prefix-caching \ --async-scheduling \ --speculative_config.model $DSPARK_CKPT \ --speculative_config.num_speculative_tokens 3 \ --mamba-backend flashinfer \ --reasoning-parser nemotron_v3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choiceMulti-GPU (TP4 with expert parallelism):
shell
vllm serve --model $MODEL_CKPT \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --enable-prefix-caching \ --async-scheduling \ --mamba-backend flashinfer \ --mamba-cache-mode align \ --reasoning-parser nemotron_v3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choiceExpert parallelism is limited to 2 or 4 ranks.
--enable-expert-parallelsplits each layer's experts across ranks, and this model's per-layer counts are 96, 100 and 116 — divisible by 4 but not by 8. Tensor parallelism without--enable-expert-parallelshards inside each expert instead and is unaffected, so TP8 remains available that way.
Context length. These snippets serve the full 1M-token context window by
default. If you are memory-constrained, or want more KV-cache headroom at high
concurrency, lower --max-model-len to match your workload.
Only plain single-GPU serving was re-validated on this checkpoint; the speculative-decoding, multi-GPU and W4A16/Ampere paths are inherited from the base model card and were not re-tested here.
| HumanEval+ | 87.20% | 84.15% | 87.80% |
| Avg. | 80.84% | 75.28% | 76.87% |
| +39% |
| concurrent 1M-token requests | 2.10× | 2.92× | +39% |
attention block size 4176 tokens to ensure that attention page size is >= mamba page sizeexport MODEL_CKPT=nota-ai/Nemotron-3.5-Lightning-30B-A3B-NVFP4-Global-Pruned-15export VLLM_PATH=/path/to/vllmcp patch/configuration_nemotron_h.py \ $($VLLM_PATH/bin/python -c "import transformers.models.nemotron_h.configuration_nemotron_h as m; print(m.__file__)")vllm serve --model $MODEL_CKPT \ --max-num-seqs 256 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --async-scheduling \ --mamba-backend flashinfer \ --mamba-cache-mode align \ --reasoning-parser nemotron_v3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choicevllm serve --model $MODEL_CKPT \ --max-num-seqs 128 \ --enable-prefix-caching \ --async-scheduling \ --speculative_config.model $DSPARK_CKPT \ --speculative_config.num_speculative_tokens 3 \ --mamba-backend flashinfer \ --reasoning-parser nemotron_v3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choicevllm serve --model $MODEL_CKPT \ --tensor-parallel-size 4 \ --enable-expert-parallel \ --enable-prefix-caching \ --async-scheduling \ --mamba-backend flashinfer \ --mamba-cache-mode align \ --reasoning-parser nemotron_v3 \ --tool-call-parser qwen3_coder \ --enable-auto-tool-choice