halt95

Nemotron-3.5-Super-VL-120B-A12B-W4A16-Merlin

README

License: openmdw-1.1

The checkpoint

Table with columns: Component, Format
Component	Format
Routed experts (512 per MoE layer)	INT4, group 128, symmetric round-to-nearest (no calibration data)
Mamba-2 mixer `in_proj` / `out_proj`	INT8 per output channel
MTP draft head: routed experts	INT4, group 128
MTP draft head: `eh_proj`, attention, latent and shared-expert projections	INT8, group 128
Main model (outside the MTP head): attention, latent projections, shared experts, routers, `conv1d`; the MTP head's routers; embeddings, `lm_head`, vision tower and projector	BF16, unchanged (about 21K small parameters stay F32, as upstream)
KV cache	FP8 E4M3, set by the engine at serve time

Formats from the quantization_config in config.json, also listed in NOTICE; the KV cache format is set by the engine profile.

Files in this repository

Table with columns: Files, What they hold, Size
Files	What they hold	Size
`model-000NN-of-00063.safetensors` (63)	Every weight above, language model, MTP head and vision tower	67.9 GiB
`model.safetensors.index.json`	Which file holds each tensor	13.9 MiB
`config.json` (it carries the quantisation config), `generation_config.json`	Model, quantisation and generation settings	8.8 KiB
`chat_template.jinja`	NVIDIA's chat template, unchanged (byte-identical to the upstream file); Merlin serves it	12.6 KiB
`tokenizer.json`, tokenizer and processor configs	Tokenizer and image-processor settings	16.3 MiB
`__init__.py`, `modeling_.py`, `configuration_.py`, `processing_.py`, `image_processing_.py`, `ultra_v3_reasoning_parser.py`	NVIDIA's model code and reasoning parser, loaded as remote code; Merlin checks it against the recipe's `remote-code.sha256` before it starts	84 KiB

About 68 GiB in all. Also in the repository: a 136-byte model-summary-idxs.safetensors (NVIDIA's two-element summary_idxs tensor for the vision encoder, unchanged, re-saved with a format: pt header; NVIDIA's file is 104 bytes), LICENSE (OpenMDW-1.1), NOTICE (every file we changed), SHA256SUMS, NVIDIA's model card (NVIDIA-MODEL-CARD.md, revision 1f8953f7, unchanged) and its bias.md, explainability.md, privacy.md and safety.md and two charts. OpenMDW-1.1 requires anyone redistributing the checkpoint to keep a copy of the licence and NVIDIA's copyright and origin notices. The weights were built from NVIDIA's revision 533d6623.

Speed

Table with columns: Prompt, Decode, one stream, Time to first token
Prompt	Decode, one stream	Time to first token
4K	101 tok/s	1.04 s
32K (3.0.1 run)	98 tok/s	6.81 s
131K	101 tok/s	27.7 s
262K	106 tok/s	62.1 s

4× RTX 3090, the needle-prompt ladder (4,269 / 34,147 / 136,625 / 261,888 prompt tokens, counted by Nemotron's tokenizer), every send a cold prefill (at 262K the three sends share one cache salt, but in this run none reused the prefix). The 32K row was not re-run for 3.0.2. Decode: thinking on (this template has no effort levels, so full thinking), ≤ 256 tokens out (128 at 262K), median of 3. Time to first token: the median below 262K; the cold send 1 at 262K. Thinking off at 262K: 86 tok/s (the three sends ran 85.0–87.1). The 262K needle is found in 3 of 3 thinking-off sends (3.0.0 run, 2026-10-20).

Table with columns: Cold prefill, one send, Prefill, Time to first token
Cold prefill, one send	Prefill	Time to first token
10,671 tokens	4,914 tok/s	2.17 s
106,725 tokens	5,111 tok/s	20.9 s
261,888 tokens	4,217 tok/s	62.1 s

One cold request per prompt, 8 tokens out at 10K and 100K. The 261,888-token row is the cold thinking-on send 1 of the exact-262K run (128 tokens out), the same send as the 262K row above.

Table with columns: Agents at once, Total tok/s, Run
Agents at once	Total tok/s	Run
2	130	3.0.0 run (pre-release build, 2026-10-17)
4	233	3.0.0 run (pre-release build, 2026-10-17)
8	432	3.0.1 run, not re-run for 3.0.2
8, thinking on	432	3.0.1 run, not re-run for 3.0.2

N 16K conversations, then N simultaneous 512-token follow-ups, thinking off unless stated, total tok/s while all N decode. The 2- and 4-agent rows were not re-run. With thinking on a 512-token follow-up may end inside the reasoning, and reasoning tokens count, so that total measures decoded tokens, not finished answers. Each overlap window was short (7.34 and 8.61 s for 2 and 4 agents; 8.89 s thinking off and 8.63 s thinking on for 8), so read these rows loosely.

Table with columns: Sessions held at once, Needles found, Re-sent from cache, lowest
Sessions held at once	Needles found	Re-sent from cache, lowest
5 × 262K	5 / 5	99.96 %
9 × 131K	9 / 9	99.91 %

N needle prompts (thinking off, greedy) sent at once, then re-sent one by one: every re-send served from the prefix cache means the pool held all N. Peak pool use 95.0 %, 0 preemptions. Nine is the engine's sequence limit for this checkpoint, not the pool's. 3.0.0 run (2026-10-20), not re-run for 3.0.1 or 3.0.2.

Table with columns: Measure, 3.0.1, 3.0.2, Change
Measure	3.0.1	3.0.2	Change
Decode, 4,269-token prompt	102.6	101.2	−1.4 %
Decode, 136,625-token prompt	101.5	100.6	−0.9 %
Decode, 261,888 tokens, thinking on	92.7	105.7	+14.0 %
Decode, 261,888 tokens, thinking off	81.8	86.3	+5.5 %
Prefill, 10,671 tokens	4,545	4,914	+8.1 %
Prefill, 106,725 tokens	4,689	5,111	+9.0 %
Prefill, 261,888 tokens	3,916	4,217	+7.7 %

tok/s. Decode: median of 3, the same frozen driver and prompts; thinking on is full thinking on both releases. Prefill: one cold send. Both releases on the same host and settings (the shipped 1M context, the optional GPU unlock). Cold time to first token at 262K: 62.1 s against 66.9 s. MTP acceptance moves decode by several percent between sends: the three 3.0.2 sends at exactly 262K with thinking on ran 99.7–108.0 tok/s, all above the 3.0.1 figure. 3.0.2 also changes this checkpoint's all-reduce with P2P on to NCCL's Simple protocol instead of LL (output identical); no A/B splits the prefill gain between that and the vLLM 0.33 base. The 32K row and the 8-agent runs were not re-run for 3.0.2. The 3.0.1 bench card has the full 3.0.1 tables.

Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02; the 3.0.1 and 3.0.2 rows also with the optional Zagitta's gpu-unlock host module.
Engine: Merlin 3.0.2 on vLLM 0.33.0, on the release candidate (the changes after it touch only Flash-Next code paths), served by merlin serve with the shipped defaults, the 1M context included, one fresh boot; measured 2026-11-04 (UTC). Rows marked 3.0.0 run or 3.0.1 run were not re-run. Decode and prefill used prompts up to 262K; speed at a 1M-token prompt has not been measured.
Layout: TP2 × PP2 + EP on four cards (layer split 46 / 42), Mamba-2 RecoverSSM, MTP with 3 draft tokens, FP8 KV, thinking on. This template has no effort levels, so thinking on is its full thinking; the bench driver's reasoning_effort medium has no effect.
Decode: T=0, a needle prompt at depth, every send a cold prefill (at 262K the three sends share one cache salt; in this run none reused the prefix); one discarded warm-up, then 3 sends per cell, median. tok/s = tokens after the first streamed event / time from first to last event.
The ladder's prompts are sized in Qwen tokens and Nemotron's tokenizer counts the same text about 4.8 % longer, so the ladder rows above are Nemotron's own counts and the 262K rows are the exact 261,888-token sends. The two single cold sends were scaled to match the Qwen checkpoints' 10K and 100K prompts in the 3.0.0 run (10,187 and 101,885 tokens); the 3.0.1 and 3.0.2 sends are 10,671 and 106,725 tokens.
Streams are client-timed. Full tables, thinking-off rows and ITL: the 3.0.2 bench card; the previous release: the 3.0.1 bench card.

Quality

This checkpoint, measured on the Merlin engine:

Table with columns: Check, Result, Setting
Check	Result	Setting
GSM8K-200, thinking on	197 / 200 (3.0.1 run)	5-shot, thinking on (full thinking); one answer hit the 8,192-token cap with no answer and counts as wrong
GSM8K-200, thinking off	195 / 200	5-shot, thinking off (3.0.0 run)
Needles at 131K	10 / 10	single-occurrence needles, an image in the prompt, thinking off, greedy, cold cache (3.0.0 run)
Needles at 262K	23 / 23	as above (3.0.0 run)
Needle at exactly 261,888 tokens	3 / 3	thinking off (3.0.0 run)
Needle at 1M (1,048,052 tokens)	1 / 1	thinking off, cold cache; 731 s to the answer (3.0.0 run)
Image, 4096 × 2048 (8,388,608 px)	1 / 1	thinking off; names both colours of a two-colour image (the processor downsizes it to about 3.4 MP); 3.0.0 run

GSM8K with thinking on (3.0.1 run): 2026-11-02, the 3.0.0 set (the same 200 problems, 5-shot, same scorer, temperature 0) on four cards. Rows marked 3.0.0 run were not re-run for 3.0.1 or 3.0.2: GSM8K with thinking off and the 131K and 262K needle panel come from the pre-release engine build of 2026-10-17; the exact-262K needle and the image from the Merlin 3.0.0 release wheel, 2026-10-20; the 1M needle from Merlin 3.0.0 at its 1M default (max_model_len 1,048,576), 2026-10-21. The two GSM8K scores differ in setting (thinking on against off), so they are not a before-and-after pair. These are checks of this checkpoint, not a matched BF16 comparison: no BF16 run of the same checks was made, so how much quality the 4-bit build keeps is not quantified here.

The base model, as reported by NVIDIA for the BF16 weights:

Table with columns: Area, Benchmark, Nemotron 3.5 Super VL (BF16)
Area	Benchmark	Nemotron 3.5 Super VL (BF16)
Reasoning and knowledge	GPQA-Diamond	89.52
Reasoning and knowledge	HLE, text-only	31.79
Coding	LiveCodeBench v6	88.22
Coding	SWE-bench Verified	71.10
Agents	Terminal-Bench 2.1	60.67
Agents	tau2-bench retail	69.24
Instructions and long context	IFBench	70.86
Instructions and long context	AA-LCR v1.1	68.62
Vision and documents	MMLongBench-Doc	62.82
Vision and documents	OCRBench v2, EN	68.12
Vision and documents	CharXiv reasoning	80.82

A selection from the benchmark table on the upstream card (its BF16 column, headed "Nemotron 3.5 SuperGA BF16"), which has the full table and comparisons with other models. These were not re-run on this 4-bit build.

What it fits on

Table with columns: Measured on, Context per request, KV pool, Held at once, Images
Measured on	Context per request	KV pool	Held at once	Images
4× RTX 3090, 24 GB (🟢 Tested)	1,048,576 tokens (1M)	1,856,367 tokens, FP8, all in VRAM	1 × 1M, 5 × 262K or 9 × 131K	One per prompt, downsized to about 3.4 MP (up to 8,388,608 px accepted)

Measured on the reference host (3.0.0 run, 2026-10-20 bench card; not re-run for 3.0.1 or 3.0.2): five 262K and nine 131K sessions held at once with every re-send served from cache; a 4096 × 2048 image with at least 686 MiB free on every card. The KV pin (2.50 GiB per card) has not had a sustained-load test yet. At the 1M default (3.0.0 run, 2026-10-21 and 22): a 1,048,052-token needle found exactly and, re-sent, served from the resident cache (1,047,936 tokens cached, 3 s to first token, no preemptions); five 262K and nine 131K needle prompts sent at once all answered; the 8 MP image left at least 666 MiB free on every card. The KV pool is 1,856,367 tokens for 2.50 GiB per card at the shipped 1M context, the figure on the 3.0.1 and 3.0.2 bench cards; the 3.0.0 bench card's 1,392,275 tokens was measured at a 262K context length. The sessions above are what was measured.

Other GPUs. The engine ships profiles for these, all on four cards; the Status column says how far each has been run:

Table with columns: GPUs, Arch, Profile, Context fit, Status
GPUs	Arch	Profile	Context fit	Status
4× RTX 3090 Ti, RTX A5000 (24 GB)	Ampere, sm_86	`svl-quad-3090`	~5 × 262K (as the 3090; needs P2P)	⚫ Untested
4× RTX 3090 without P2P	Ampere, sm_86	`svl-quad-3090-nop2p`	~5 × 262K (the 3090 pin; P2P off ran on 4× RTX 4090)	⚫ Untested
4× RTX 4090 (24 GB)	Ada, sm_89	`svl-quad-sm89-24g`	5 × 262K (rented host)	🟡 Testing
4× 48 GB: RTX A6000, A40	Ampere, sm_86	`svl-quad-sm86-48g`	~54 × 262K (32 at once)	⚫ Untested
4× 32 / 48 GB Ada: RTX 4080 Super (modified), RTX 4090 48 GB (modified), RTX 6000 Ada, L40S	Ada, sm_89	`svl-quad-sm89-32g`, `svl-quad-sm89-48g`	~5 × 262K (the 24 GB pin)	⚫ Untested
4× A30 (24 GB)	Ampere, sm_80	`svl-quad-sm80-24g`	~5 × 262K	⚫ Untested
4× A100 40 / 80 GB	Ampere, sm_80	`svl-quad-sm80-40g`, `svl-quad-sm80-80g`	~37 / ~119 × 262K (32 at once)	⚫ Untested
4× CMP 170HX 32 GB	Ampere, sm_80	`svl-quad-sm80-32g`	~21 × 262K (32 at once)	⚫ Untested
4× CMP 170HX 64 GB	Ampere, sm_80	`svl-quad-sm80-64g`	~86 × 262K (32 at once; 9 × 262K needles exact on a rented host)	🟡 Testing

merlin serve and ./setup.sh pick the profile from the GPUs they find. A ~ is the engine's estimate, not a measurement. Context fit is what the KV pool holds; the server admits at most 9 requests at once on 4× RTX 3090, RTX 3090 Ti and RTX A5000 with P2P (svl-quad-3090), 10 on the other rows with a 5 × 262K pool (including 4× RTX 3090 without P2P), and 32 on the larger-pool rows (each profile's sequence limit); the rest queue. Every profile uses four cards and an FP8 KV cache, on sm_80 cards too (FP8 KV ran on the 4× CMP 170HX 64 GB rental, needles exact). 🟢 measured by us · 🟡 testing: full-model runs passed on rented, cloud-hosted cards, kernel parity qualification still to run · ⚫ a profile ships, not yet qualified by us (expected to work; a rented-host run is noted in the row where one was made). Every profile: the engine's hardware matrix.

Table with columns: Need, Details
Need	Details
Host RAM	Not yet measured for this checkpoint.
Disk	About 68 GiB for the checkpoint, plus the engine environment and caches
Engine host	Linux x86-64 (glibc 2.38 or newer), NVIDIA driver R580 or later (CUDA 13), Python 3.13, GCC/G++ 13 or newer, `/dev/shm` of at least 1 GB; GPUs headless with no other CUDA process. `merlin doctor` checks the host.

Host requirements are the engine's; disk is this repository's size.

Use it

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine./setup.sh --model nemotron-super-vl

One command does the rest: after a small uv and Python bootstrap it checks the GPUs, driver, RAM and disk, shows the engine, package and checkpoint downloads with their sizes before fetching them, installs the prebuilt engine, downloads and verifies this checkpoint, and starts an OpenAI-compatible server on 127.0.0.1:8000. Enter takes the recommended answer at each question; --yes takes them all, --dry-run shows the plan. Run ./setup.sh again to start the server, and git pull && ./setup.sh to update the engine without downloading the model again. Requests use the model name svl.

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-enginecurl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.2/vllm-0.33.0+merlin.665f43d2cf-cp313-cp313-linux_x86_64.whlMERLIN_WHEEL="$PWD/vllm-0.33.0+merlin.665f43d2cf-cp313-cp313-linux_x86_64.whl" bash release/install-env.sh ./venv venv/bin/hf download halt95/Nemotron-3.5-Super-VL-120B-A12B-W4A16-Merlin --revision a0c5c92c01ac87bee8fd94299897970bc83072bf \  --local-dir ~/models/nemotron-super-vl(cd ~/models/nemotron-super-vl && sha256sum -c SHA256SUMS)venv/bin/python merlin_cli.py serve ~/models/nemotron-super-vl

MERLIN_WHEEL installs the prebuilt engine (sm_80, sm_86, sm_89; Merlin 3.0.2 on vLLM 0.33.0, a new wheel that needs a new venv), checked against release/merlin-wheel.sha256; without it the installer compiles the engine instead, about 1.5 hours on 8 cores. merlin serve recognises the checkpoint, picks the largest tested layout for the GPUs it finds (or, with none, the largest untested one that fits, labelled as such) and serves on 127.0.0.1:8000. Add --dry-run to print the resolved command without starting anything. The first start compiles kernels; the server is ready when merlin: Ready: ... prints.

Recommended settings (NVIDIA's, for this model):

Table with columns: Input, temperature, top_p
Input	temperature	top_p
Text	1.0	0.95
Images	0.6	0.95

NVIDIA's recommendation on the upstream card. Reasoning tokens and the answer share the max_tokens budget.

Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
Reasoning levels are NVIDIA's own: off, on, and on (medium) with "chat_template_kwargs": {"enable_thinking": true, "medium_effort": true}, which writes fewer reasoning tokens and, NVIDIA notes, may lower accuracy. Merlin serves full thinking by default, and every thinking-on figure on this card is full thinking; reasoning_effort (the Qwen checkpoints' switch) has no effect on this model, whose template has no such effort levels.
Chat template: Merlin serves the checkpoint's own Nemotron template, with the nemotron_v3 reasoning parser and the qwen3_coder tool-call parser, the way NVIDIA's card serves it. The Froggeric template the Qwen checkpoints use does not apply.
One image per prompt, downsized by the model's processor to at most 13,312 patches (about 3.4 MP); images up to the engine's admission limit of 8,388,608 px are accepted. Video is off in Merlin's launcher.

The base model

Nemotron 3.5 Super VL is NVIDIA's vision-language model for reasoning, coding, tool use, documents and video. The architecture, in short:

Hybrid layers. Mamba-2 state-space layers, Mixture-of-Experts layers and a few attention layers, interleaved: the Mamba-2 layers keep long-context cost down.
Latent experts. 512 routed experts per MoE layer, 22 active plus 1 shared per token, run through a 1,024-wide latent projection: 12B of the 120B parameters do the work for each token.
Vision. A C-RADIO encoder and a vision-language connector take images, document pages and video frames.
Multi-token prediction. A shared MTP head trained for speculative decoding; Merlin uses it with 3 draft tokens.
Reasoning levels set in the chat template: off, on, and on (medium).

Table with columns: Property, Nemotron 3.5 Super VL
Property	Nemotron 3.5 Super VL
Type	Mamba-2–Transformer hybrid MoE with a C-RADIO vision encoder (text, images and video in; text out)
Parameters	120B total, 12B active
Layers	88: 40 Mamba-2, 40 MoE, 8 attention; plus one MTP layer (attention + MoE)
Experts	512 routed per MoE layer; 22 routed + 1 shared active; latent width 1,024
Attention	32 query / 2 KV heads, head dim 128 · Mamba-2: 128 heads, head dim 64, state 128
Context	1M tokens (NVIDIA); this build serves the full 1,048,576

Architecture as reported by NVIDIA on the upstream card; layer and head counts from its config.json.

Other builds

Table with columns: Checkpoint, Base, GPUs, Decode at 262K
Checkpoint	Base	GPUs	Decode at 262K
Nemotron 3.5 Super VL (this card)	Nemotron 3.5 Super VL, 120B-A12B MoE	4× 3090	106 tok/s
Flash-Next	Qwen3.8-Flash-Next, 125B MoE	4× 3090	207 tok/s
Swift1.5	Flash-Next fine-tune	4× 3090	186 tok/s
Swift1.5-Huihui	Swift1.5, abliterated	4× 3090	193 tok/s
Qwen3.8-27B	Qwen3.8-27B, dense	1×, 2× or 4× 3090	77 / 146 / 250 tok/s
ThinkingCap	Qwen3.8-27B fine-tune	2× or 4× 3090	160 / 223 tok/s
Ornith 1.5	Qwen3.5-122B-A10B fine-tune (ornith-ai), 122B MoE	4× 3090	111 tok/s

One stream, thinking on, 261,888 prompt tokens + 128 out, median of 3; Merlin 3.0.2, 2026-11-04 and 2026-11-05 bench cards. Thinking on at reasoning effort medium, the server default, wherever the checkpoint has effort levels; the Nemotron and Ornith 1.5 templates have none, so those two ran at full thinking.

Good to know

Every figure we measured for this checkpoint comes from 4× RTX 3090 at 220 W, PCIe Gen4 with P2P (the 3.0.1 and 3.0.2 speed rows also with the optional Zagitta's gpu-unlock host module), except the rental rows under Other GPUs; the Other builds table names each build's card count. Hosts without P2P use the no-P2P profiles (see What it fits on).
Four cards are the minimum: two-card hosts are refused.
Independent derivative: not made, reviewed or endorsed by NVIDIA. NVIDIA's model card (NVIDIA-MODEL-CARD.md, revision 1f8953f7) covers training data, intended use, evaluation and limitations; read it before use.

Changes

Table with columns: Date, Change to this repository
Date	Change to this repository
2026-10-16	The checkpoint as shipped: MTP head routed experts re-packed to INT4 (`config.json`, the index and shards 62 and 63 rewritten)
2026-11-02	No change to the weights, config or tokenizer. The card now quotes the Merlin 3.0.1 bench card
2026-11-05	No change to the weights, config or tokenizer. The card now quotes the Merlin 3.0.2 bench card

Built on the reference host; the Hub repository starts with this build. Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.

Credits and licence

NVIDIA (model; OpenMDW-1.1, see LICENSE; NVIDIA's model card kept unchanged as NVIDIA-MODEL-CARD.md), the vLLM project (the engine Merlin is built on), aikitoria (peer-to-peer on consumer Ampere), Neiko2002, and CoolShades (the idle-power report behind the v2.5.2 engine fix). Upstream fixes are credited in the engine repository.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

halt95

Model Tree

Base

nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16

Quantized

this model

Input Modalities

TextAudioImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: openmdw-1.1

The checkpoint

Table with columns: Component, Format
Component	Format
Routed experts (512 per MoE layer)	INT4, group 128, symmetric round-to-nearest (no calibration data)
Mamba-2 mixer `in_proj` / `out_proj`	INT8 per output channel
MTP draft head: routed experts	INT4, group 128
MTP draft head: `eh_proj`, attention, latent and shared-expert projections	INT8, group 128
Main model (outside the MTP head): attention, latent projections, shared experts, routers, `conv1d`; the MTP head's routers; embeddings, `lm_head`, vision tower and projector	BF16, unchanged (about 21K small parameters stay F32, as upstream)
KV cache	FP8 E4M3, set by the engine at serve time

Formats from the quantization_config in config.json, also listed in NOTICE; the KV cache format is set by the engine profile.

Files in this repository

Table with columns: Files, What they hold, Size
Files	What they hold	Size
`model-000NN-of-00063.safetensors` (63)	Every weight above, language model, MTP head and vision tower	67.9 GiB
`model.safetensors.index.json`	Which file holds each tensor	13.9 MiB
`config.json` (it carries the quantisation config), `generation_config.json`	Model, quantisation and generation settings	8.8 KiB
`chat_template.jinja`	NVIDIA's chat template, unchanged (byte-identical to the upstream file); Merlin serves it	12.6 KiB
`tokenizer.json`, tokenizer and processor configs	Tokenizer and image-processor settings	16.3 MiB
`__init__.py`, `modeling_.py`, `configuration_.py`, `processing_.py`, `image_processing_.py`, `ultra_v3_reasoning_parser.py`	NVIDIA's model code and reasoning parser, loaded as remote code; Merlin checks it against the recipe's `remote-code.sha256` before it starts	84 KiB

Speed

Table with columns: Prompt, Decode, one stream, Time to first token
Prompt	Decode, one stream	Time to first token
4K	101 tok/s	1.04 s
32K (3.0.1 run)	98 tok/s	6.81 s
131K	101 tok/s	27.7 s
262K	106 tok/s	62.1 s

Table with columns: Cold prefill, one send, Prefill, Time to first token
Cold prefill, one send	Prefill	Time to first token
10,671 tokens	4,914 tok/s	2.17 s
106,725 tokens	5,111 tok/s	20.9 s
261,888 tokens	4,217 tok/s	62.1 s

One cold request per prompt, 8 tokens out at 10K and 100K. The 261,888-token row is the cold thinking-on send 1 of the exact-262K run (128 tokens out), the same send as the 262K row above.

Table with columns: Agents at once, Total tok/s, Run
Agents at once	Total tok/s	Run
2	130	3.0.0 run (pre-release build, 2026-10-17)
4	233	3.0.0 run (pre-release build, 2026-10-17)
8	432	3.0.1 run, not re-run for 3.0.2
8, thinking on	432	3.0.1 run, not re-run for 3.0.2

Table with columns: Sessions held at once, Needles found, Re-sent from cache, lowest
Sessions held at once	Needles found	Re-sent from cache, lowest
5 × 262K	5 / 5	99.96 %
9 × 131K	9 / 9	99.91 %

Table with columns: Measure, 3.0.1, 3.0.2, Change
Measure	3.0.1	3.0.2	Change
Decode, 4,269-token prompt	102.6	101.2	−1.4 %
Decode, 136,625-token prompt	101.5	100.6	−0.9 %
Decode, 261,888 tokens, thinking on	92.7	105.7	+14.0 %
Decode, 261,888 tokens, thinking off	81.8	86.3	+5.5 %
Prefill, 10,671 tokens	4,545	4,914	+8.1 %
Prefill, 106,725 tokens	4,689	5,111	+9.0 %
Prefill, 261,888 tokens	3,916	4,217	+7.7 %

Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02; the 3.0.1 and 3.0.2 rows also with the optional Zagitta's gpu-unlock host module.
Engine: Merlin 3.0.2 on vLLM 0.33.0, on the release candidate (the changes after it touch only Flash-Next code paths), served by merlin serve with the shipped defaults, the 1M context included, one fresh boot; measured 2026-11-04 (UTC). Rows marked 3.0.0 run or 3.0.1 run were not re-run. Decode and prefill used prompts up to 262K; speed at a 1M-token prompt has not been measured.
Layout: TP2 × PP2 + EP on four cards (layer split 46 / 42), Mamba-2 RecoverSSM, MTP with 3 draft tokens, FP8 KV, thinking on. This template has no effort levels, so thinking on is its full thinking; the bench driver's reasoning_effort medium has no effect.
Decode: T=0, a needle prompt at depth, every send a cold prefill (at 262K the three sends share one cache salt; in this run none reused the prefix); one discarded warm-up, then 3 sends per cell, median. tok/s = tokens after the first streamed event / time from first to last event.
The ladder's prompts are sized in Qwen tokens and Nemotron's tokenizer counts the same text about 4.8 % longer, so the ladder rows above are Nemotron's own counts and the 262K rows are the exact 261,888-token sends. The two single cold sends were scaled to match the Qwen checkpoints' 10K and 100K prompts in the 3.0.0 run (10,187 and 101,885 tokens); the 3.0.1 and 3.0.2 sends are 10,671 and 106,725 tokens.
Streams are client-timed. Full tables, thinking-off rows and ITL: the 3.0.2 bench card; the previous release: the 3.0.1 bench card.

Quality

This checkpoint, measured on the Merlin engine:

Table with columns: Check, Result, Setting
Check	Result	Setting
GSM8K-200, thinking on	197 / 200 (3.0.1 run)	5-shot, thinking on (full thinking); one answer hit the 8,192-token cap with no answer and counts as wrong
GSM8K-200, thinking off	195 / 200	5-shot, thinking off (3.0.0 run)
Needles at 131K	10 / 10	single-occurrence needles, an image in the prompt, thinking off, greedy, cold cache (3.0.0 run)
Needles at 262K	23 / 23	as above (3.0.0 run)
Needle at exactly 261,888 tokens	3 / 3	thinking off (3.0.0 run)
Needle at 1M (1,048,052 tokens)	1 / 1	thinking off, cold cache; 731 s to the answer (3.0.0 run)
Image, 4096 × 2048 (8,388,608 px)	1 / 1	thinking off; names both colours of a two-colour image (the processor downsizes it to about 3.4 MP); 3.0.0 run

The base model, as reported by NVIDIA for the BF16 weights:

Table with columns: Area, Benchmark, Nemotron 3.5 Super VL (BF16)
Area	Benchmark	Nemotron 3.5 Super VL (BF16)
Reasoning and knowledge	GPQA-Diamond	89.52
Reasoning and knowledge	HLE, text-only	31.79
Coding	LiveCodeBench v6	88.22
Coding	SWE-bench Verified	71.10
Agents	Terminal-Bench 2.1	60.67
Agents	tau2-bench retail	69.24
Instructions and long context	IFBench	70.86
Instructions and long context	AA-LCR v1.1	68.62
Vision and documents	MMLongBench-Doc	62.82
Vision and documents	OCRBench v2, EN	68.12
Vision and documents	CharXiv reasoning	80.82

What it fits on

Table with columns: Measured on, Context per request, KV pool, Held at once, Images
Measured on	Context per request	KV pool	Held at once	Images
4× RTX 3090, 24 GB (🟢 Tested)	1,048,576 tokens (1M)	1,856,367 tokens, FP8, all in VRAM	1 × 1M, 5 × 262K or 9 × 131K	One per prompt, downsized to about 3.4 MP (up to 8,388,608 px accepted)

Other GPUs. The engine ships profiles for these, all on four cards; the Status column says how far each has been run:

Table with columns: GPUs, Arch, Profile, Context fit, Status
GPUs	Arch	Profile	Context fit	Status
4× RTX 3090 Ti, RTX A5000 (24 GB)	Ampere, sm_86	`svl-quad-3090`	~5 × 262K (as the 3090; needs P2P)	⚫ Untested
4× RTX 3090 without P2P	Ampere, sm_86	`svl-quad-3090-nop2p`	~5 × 262K (the 3090 pin; P2P off ran on 4× RTX 4090)	⚫ Untested
4× RTX 4090 (24 GB)	Ada, sm_89	`svl-quad-sm89-24g`	5 × 262K (rented host)	🟡 Testing
4× 48 GB: RTX A6000, A40	Ampere, sm_86	`svl-quad-sm86-48g`	~54 × 262K (32 at once)	⚫ Untested
4× 32 / 48 GB Ada: RTX 4080 Super (modified), RTX 4090 48 GB (modified), RTX 6000 Ada, L40S	Ada, sm_89	`svl-quad-sm89-32g`, `svl-quad-sm89-48g`	~5 × 262K (the 24 GB pin)	⚫ Untested
4× A30 (24 GB)	Ampere, sm_80	`svl-quad-sm80-24g`	~5 × 262K	⚫ Untested
4× A100 40 / 80 GB	Ampere, sm_80	`svl-quad-sm80-40g`, `svl-quad-sm80-80g`	~37 / ~119 × 262K (32 at once)	⚫ Untested
4× CMP 170HX 32 GB	Ampere, sm_80	`svl-quad-sm80-32g`	~21 × 262K (32 at once)	⚫ Untested
4× CMP 170HX 64 GB	Ampere, sm_80	`svl-quad-sm80-64g`	~86 × 262K (32 at once; 9 × 262K needles exact on a rented host)	🟡 Testing

Table with columns: Need, Details
Need	Details
Host RAM	Not yet measured for this checkpoint.
Disk	About 68 GiB for the checkpoint, plus the engine environment and caches
Engine host	Linux x86-64 (glibc 2.38 or newer), NVIDIA driver R580 or later (CUDA 13), Python 3.13, GCC/G++ 13 or newer, `/dev/shm` of at least 1 GB; GPUs headless with no other CUDA process. `merlin doctor` checks the host.

Host requirements are the engine's; disk is this repository's size.

Use it

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine./setup.sh --model nemotron-super-vl

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine./setup.sh --model nemotron-super-vl

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-enginecurl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.2/vllm-0.33.0+merlin.665f43d2cf-cp313-cp313-linux_x86_64.whlMERLIN_WHEEL="$PWD/vllm-0.33.0+merlin.665f43d2cf-cp313-cp313-linux_x86_64.whl" bash release/install-env.sh ./venv venv/bin/hf download halt95/Nemotron-3.5-Super-VL-120B-A12B-W4A16-Merlin --revision a0c5c92c01ac87bee8fd94299897970bc83072bf \  --local-dir ~/models/nemotron-super-vl(cd ~/models/nemotron-super-vl && sha256sum -c SHA256SUMS)venv/bin/python merlin_cli.py serve ~/models/nemotron-super-vl

git clone https://github.com/halt95/merlin-engine.git && cd merlin-enginecurl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.2/vllm-0.33.0+merlin.665f43d2cf-cp313-cp313-linux_x86_64.whlMERLIN_WHEEL="$PWD/vllm-0.33.0+merlin.665f43d2cf-cp313-cp313-linux_x86_64.whl" bash release/install-env.sh ./venv venv/bin/hf download halt95/Nemotron-3.5-Super-VL-120B-A12B-W4A16-Merlin --revision a0c5c92c01ac87bee8fd94299897970bc83072bf \  --local-dir ~/models/nemotron-super-vl(cd ~/models/nemotron-super-vl && sha256sum -c SHA256SUMS)venv/bin/python merlin_cli.py serve ~/models/nemotron-super-vl

Recommended settings (NVIDIA's, for this model):

Table with columns: Input, temperature, top_p
Input	temperature	top_p
Text	1.0	0.95
Images	0.6	0.95

NVIDIA's recommendation on the upstream card. Reasoning tokens and the answer share the max_tokens budget.

Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
Reasoning levels are NVIDIA's own: off, on, and on (medium) with "chat_template_kwargs": {"enable_thinking": true, "medium_effort": true}, which writes fewer reasoning tokens and, NVIDIA notes, may lower accuracy. Merlin serves full thinking by default, and every thinking-on figure on this card is full thinking; reasoning_effort (the Qwen checkpoints' switch) has no effect on this model, whose template has no such effort levels.
Chat template: Merlin serves the checkpoint's own Nemotron template, with the nemotron_v3 reasoning parser and the qwen3_coder tool-call parser, the way NVIDIA's card serves it. The Froggeric template the Qwen checkpoints use does not apply.
One image per prompt, downsized by the model's processor to at most 13,312 patches (about 3.4 MP); images up to the engine's admission limit of 8,388,608 px are accepted. Video is off in Merlin's launcher.

The base model

Nemotron 3.5 Super VL is NVIDIA's vision-language model for reasoning, coding, tool use, documents and video. The architecture, in short:

Hybrid layers. Mamba-2 state-space layers, Mixture-of-Experts layers and a few attention layers, interleaved: the Mamba-2 layers keep long-context cost down.
Latent experts. 512 routed experts per MoE layer, 22 active plus 1 shared per token, run through a 1,024-wide latent projection: 12B of the 120B parameters do the work for each token.
Vision. A C-RADIO encoder and a vision-language connector take images, document pages and video frames.
Multi-token prediction. A shared MTP head trained for speculative decoding; Merlin uses it with 3 draft tokens.
Reasoning levels set in the chat template: off, on, and on (medium).

Table with columns: Property, Nemotron 3.5 Super VL
Property	Nemotron 3.5 Super VL
Type	Mamba-2–Transformer hybrid MoE with a C-RADIO vision encoder (text, images and video in; text out)
Parameters	120B total, 12B active
Layers	88: 40 Mamba-2, 40 MoE, 8 attention; plus one MTP layer (attention + MoE)
Experts	512 routed per MoE layer; 22 routed + 1 shared active; latent width 1,024
Attention	32 query / 2 KV heads, head dim 128 · Mamba-2: 128 heads, head dim 64, state 128
Context	1M tokens (NVIDIA); this build serves the full 1,048,576

Architecture as reported by NVIDIA on the upstream card; layer and head counts from its config.json.

Other builds

Table with columns: Checkpoint, Base, GPUs, Decode at 262K
Checkpoint	Base	GPUs	Decode at 262K
Nemotron 3.5 Super VL (this card)	Nemotron 3.5 Super VL, 120B-A12B MoE	4× 3090	106 tok/s
Flash-Next	Qwen3.8-Flash-Next, 125B MoE	4× 3090	207 tok/s
Swift1.5	Flash-Next fine-tune	4× 3090	186 tok/s
Swift1.5-Huihui	Swift1.5, abliterated	4× 3090	193 tok/s
Qwen3.8-27B	Qwen3.8-27B, dense	1×, 2× or 4× 3090	77 / 146 / 250 tok/s
ThinkingCap	Qwen3.8-27B fine-tune	2× or 4× 3090	160 / 223 tok/s
Ornith 1.5	Qwen3.5-122B-A10B fine-tune (ornith-ai), 122B MoE	4× 3090	111 tok/s

Good to know

Every figure we measured for this checkpoint comes from 4× RTX 3090 at 220 W, PCIe Gen4 with P2P (the 3.0.1 and 3.0.2 speed rows also with the optional Zagitta's gpu-unlock host module), except the rental rows under Other GPUs; the Other builds table names each build's card count. Hosts without P2P use the no-P2P profiles (see What it fits on).
Four cards are the minimum: two-card hosts are refused.
Independent derivative: not made, reviewed or endorsed by NVIDIA. NVIDIA's model card (NVIDIA-MODEL-CARD.md, revision 1f8953f7) covers training data, intended use, evaluation and limitations; read it before use.

Changes

Table with columns: Date, Change to this repository
Date	Change to this repository
2026-10-16	The checkpoint as shipped: MTP head routed experts re-packed to INT4 (`config.json`, the index and shards 62 and 63 rewritten)
2026-11-02	No change to the weights, config or tokenizer. The card now quotes the Merlin 3.0.1 bench card
2026-11-05	No change to the weights, config or tokenizer. The card now quotes the Merlin 3.0.2 bench card

Built on the reference host; the Hub repository starts with this build. Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.