Installation
vLLM needs a small patch so that MoE layers may have different numbers of experts.
uv venv --python 3.12 venvsource venv/bin/activateuv pip install vllm==0.31.0 cd patchpython apply_patch.py --dry-run # show the changespython apply_patch.py # apply the patch (originals are backed up to vllm/.nota_patch_backup/)python apply_patch.py --revert # restore the original files
After applying the patch, serve the model on 1 x B200 with the arguments of the
original model card (1 x GB200 recipe: MTP, reasoning parser, tool-call parser):
vllm serve nota-ai/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4-Global-Pruned-20 \ --served-model-name nemotron-3.5-super-vl \ --tensor-parallel-size 1 \ --trust-remote-code \ --max-num-seqs 512 \ --enable-prefix-caching \ --max-num-batched-tokens 32768 \ --prefix-match-unit 16 \ --enable-mamba-shared-prefix-checkpoint \ --async-scheduling \ --kernel-config '{"linear_backend_per_quant":{"fp8_w8a8":"torch"}}' \ --compilation-config '{"pass_config":{"fuse_attn_quant":true},"use_inductor_graph_partition":true,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \ --mamba-backend flashinfer \ --mamba-ssm-cache-dtype float16 \ --enable-mamba-cache-stochastic-rounding \ --mamba-cache-philox-rounds 5 \ --mamba-cache-mode align \ --video-pruning-rate 0.5 \ --media-io-kwargs '{"video": {"fps": 2, "num_frames": 256}}' \ --enable-auto-tool-choice \ --speculative_config.method mtp \ --speculative_config.num_speculative_tokens 5 \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3
Reasoning is switched per request with chat_template_kwargs (e.g. {"enable_thinking": false}); see the
original model card for API client examples.
Main results
Table with columns: Model, Pruning ratio, English avg. accuracy, Korean avg. accuracy, Korean language consistency| Model | Pruning ratio | English avg. accuracy | Korean avg. accuracy | Korean language consistency |
|---|
| Original (NVFP4) | - | 84.0 | 71.4 | 76.2 |
| Uniform pruning | 20% | 82.3 | 68.2 | 75.9 |
| Ours | 20% | 83.6 | 72.9 | 91.6 |
- English: IFEval, IFBench, GPQA-Diamond, AIME 2026, LiveCodeBench v6
- Korean: BenchMAX IFEval-ko, IFBench-ko, KoBALT, CLIcK
- Korean language consistency: share of Korean queries answered in Korean, averaged over the Korean benchmarks
- Uniform pruning: the same number of experts removed from every layer
Ablation (BenchMAX IFEval-ko, 20% pruned)
Table with columns: Configuration, Accuracy, Korean language consistency| Configuration | Accuracy | Korean language consistency |
|---|
| Original (NVFP4) | 73.9 | 39.9 |
| Cluster-based non-uniform expert pruning | 73.7 | 41.7 |
| + Language-Following Expert Preservation | 82.3 | 51.7 |
| + Language-Following Weight Editing (this model) | 90.9 | 87.4 |
License
This model is a modified version of nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 and is distributed under the
same license, the OpenMDW License Agreement, version 1.1. See the original model
card for the base model's intended use, limitations, bias, safety and privacy notes.