Highlights
- 2.04x Faster 32k Long-Context Pure Decode & 46,853 tok/s Batched Prefill: On the 32k Wikipedia benchmark (
25,258 prompt tokens/req) on DGX Spark GB10, NVFP4 achieves 120.9 tok/s pure decode vs. 59.2 tok/s for Base BF16 at c=1, and scales to 491.5 tok/s pure decode and 46,853.4 tok/s prompt prefill at c=32 where Base BF16 runs out of memory.
- 65% Smaller VRAM Footprint (
5.20 GiB vs. 15.20 GiB Base BF16): Packs all 42 transformer layers into NVFP4, accelerates FP8/NVFP4 embedding lookups via single-launch Triton kernels, and serves lm_head via a 4-bit B12xBlockscaledLinear (W4A16) kernel, saving 10.0 GiB of runtime VRAM over the unquantized BF16 model.
- Near-Lossless Text Accuracy (
79.09% vs. 79.40% Base BF16 — -0.31% Delta): Retains 99.6% of the unquantized 14.89 GiB BF16 baseline across 20,565 questions spanning 16 math, coding, reasoning, and instruction-following benchmarks.
- Full Omni-Modal Vision & Audio Support (
73.81% Composite): Re-enables Gemma4ForConditionalGeneration multimodal inference with MXFP8 vision and audio encoders, matching the unquantized Google Gemma 4 E4B-IT BF16 reference (73.40%).
Benchmark Results
Language (20,565 Questions)
Vision & Audio (2,203 Questions)
Inference Speed & Concurrency Scaling (NVIDIA DGX Spark GB10 & RTX 5090)
Quickstart (vLLM)
Stock vllm/vllm-openai:latest does not yet wire ModelOptMixedPrecisionConfig into VocabParallelEmbedding (embed_tokens / embed_tokens_per_layer) or ParallelLMHead. The included runtime hook (vllm_nemotron_patch.py v5) registers fused Triton FP8/NVFP4 embedding kernels and serves lm_head as a 4-bit B12xBlockscaledLinear (NVFP4 W4A16) operator. All commands below expose the short model alias bolt (--served-model-name bolt).
Option 1: Docker (vllm/vllm-openai:latest + b12x)
export BOLT_REPO="arvedo/Nemotron-3.5-Bolt-E4B-NVFP4-VisionAudio" sudo docker rm -f bolt 2>/dev/null || truesudo docker run -d --name bolt \ --gpus all \ --network host \ --ipc host \ --entrypoint /bin/bash \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ -v "$HOME/.cache/vllm:/root/.cache/vllm" \ -v "$HOME/.cache/flashinfer:/root/.cache/flashinfer" \ -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \ -e PYTHONWARNINGS="ignore::DeprecationWarning,ignore::UserWarning" \ vllm/vllm-openai:latest \ -c "pip install -q --root-user-action=ignore 'vllm[b12x]' && \ curl -sL https://huggingface.co/$BOLT_REPO/resolve/main/vllm_nemotron_patch.py | python3 - && \ exec vllm serve $BOLT_REPO \ --served-model-name bolt \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 \ --allowed-origins '*' \ --dtype bfloat16 \ --kv-cache-dtype fp8 \ --linear-backend b12x \ --attention-backend FLASHINFER \ --max-model-len 131072 \ --max-num-seqs 64 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --enable-chunked-prefill \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --reasoning-parser gemma4 \ --gpu-memory-utilization 0.80"
Option 2: DGX Spark Container (eugr/spark-vllm-b12x:latest)
export BOLT_REPO="arvedo/Nemotron-3.5-Bolt-E4B-NVFP4-VisionAudio" sudo docker rm -f bolt 2>/dev/null || truesudo docker run -d --name bolt \ --gpus all \ --network host \ --ipc host \ --entrypoint /bin/bash \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ -v "$HOME/.cache/vllm:/root/.cache/vllm" \ -v "$HOME/.cache/flashinfer:/root/.cache/flashinfer" \ -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \ eugr/spark-vllm-b12x:latest \ -c "curl -sL https://huggingface.co/$BOLT_REPO/resolve/main/vllm_nemotron_patch.py | python3 - && \ exec vllm serve $BOLT_REPO \ --served-model-name bolt \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 \ --dtype bfloat16 \ --kv-cache-dtype fp8 \ --linear-backend b12x \ --attention-backend FLASHINFER \ --max-model-len 131072 \ --max-num-seqs 64 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --enable-chunked-prefill \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --reasoning-parser gemma4 \ --gpu-memory-utilization 0.80"
Option 3: DGX Spark / High-Throughput Serving with NVIDIA DFlash NVFP4 Drafter (+50% Decode Speedup)
Accelerate token generation with NVIDIA's official NVFP4 DFlash speculative decoding draft model (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4). Because vision and audio features are completely projected during prompt prefill and decoding is pure auto-regressive text token generation, the official DFlash drafter works 100% out of the box with zero quality loss:
export BOLT_REPO="arvedo/Nemotron-3.5-Bolt-E4B-NVFP4-VisionAudio" sudo docker rm -f bolt 2>/dev/null || truesudo docker run -d --name bolt \ --gpus all \ --network host \ --ipc host \ --entrypoint /bin/bash \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ -v "$HOME/.cache/vllm:/root/.cache/vllm" \ -v "$HOME/.cache/flashinfer:/root/.cache/flashinfer" \ -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \ eugr/spark-vllm-b12x:latest \ -c "curl -sL https://huggingface.co/$BOLT_REPO/resolve/main/vllm_nemotron_patch.py | python3 - && \ exec vllm serve $BOLT_REPO \ --served-model-name bolt \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 \ --dtype bfloat16 \ --kv-cache-dtype fp8 \ --linear-backend b12x \ --attention-backend FLASHINFER \ --speculative-config '{"model": "nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4", "method": "dflash", "num_speculative_tokens": 7}' \ --max-model-len 131072 \ --max-num-seqs 64 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --enable-chunked-prefill \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --reasoning-parser gemma4 \ --gpu-memory-utilization 0.80"
Option 3: DGX Spark / High-Throughput Serving with NVIDIA DFlash NVFP4 Drafter (+50% Decode Speedup)
Accelerate token generation with NVIDIA's official NVFP4 DFlash speculative decoding draft model (nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4). Because vision and audio features are completely projected during prompt prefill and decoding is pure auto-regressive text token generation, the official DFlash drafter works 100% out of the box with zero quality loss:
export BOLT_REPO="arvedo/Nemotron-3.5-Bolt-E4B-NVFP4-VisionAudio-LowMemory" sudo docker rm -f bolt 2>/dev/null || truesudo docker run -d --name bolt \ --gpus all \ --network host \ --ipc host \ --entrypoint /bin/bash \ -v "$HOME/.cache/huggingface:/root/.cache/huggingface" \ -v "$HOME/.cache/vllm:/root/.cache/vllm" \ -v "$HOME/.cache/flashinfer:/root/.cache/flashinfer" \ -e VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 \ eugr/spark-vllm-b12x:latest \ -c "curl -sL https://huggingface.co/$BOLT_REPO/resolve/main/vllm_nemotron_patch.py | python3 - && \ exec vllm serve $BOLT_REPO \ --served-model-name bolt \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 \ --dtype bfloat16 \ --kv-cache-dtype fp8 \ --linear-backend b12x \ --attention-backend FLASHINFER \ --speculative-config '{"model": "nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4", "method": "dflash", "num_speculative_tokens": 7}' \ --max-model-len 131072 \ --max-num-seqs 64 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --enable-chunked-prefill \ --enable-auto-tool-choice \ --tool-call-parser gemma4 \ --reasoning-parser gemma4 \ --gpu-memory-utilization 0.80"
OpenAI Client Query (model="bolt")
from openai import OpenAI client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY") resp = client.chat.completions.create( model="bolt", messages=[{"role": "user", "content": "Explain NVFP4 quantization on NVIDIA Blackwell in three sentences."}], temperature=0.0,)print(resp.choices[0].message.content)
NVIDIA Nemotron-3.5-Bolt Runtime Features
In nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, NVIDIA introduced several specialized runtime controls beyond standard Gemma-4—including per-request Reasoning On/Off toggles, a concise Conversational Mode for spoken/interactive agents, Runtime Thinking Budget Control via two-step continuation, and Synchronous & Asynchronous Tool Calling (--enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4). For full upstream documentation, see the official NVIDIA model card.
Recommended upstream sampling settings are temperature=1.0 and top_p=0.95 in both thinking-enabled (default) and thinking-disabled modes:
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")MODEL = "bolt" # Reasoning On (Default)response = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "Write a haiku about GPUs"}], temperature=1.0, top_p=0.95, extra_body={ "skip_special_tokens": False, "chat_template_kwargs": {"enable_thinking": True}, },)print(response.choices[0].message.content) # Reasoning Off (Direct Answer)response = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "What is the capital of Japan?"}], temperature=1.0, top_p=0.95, extra_body={ "skip_special_tokens": False, "chat_template_kwargs": {"enable_thinking": False}, },)print(response.choices[0].message.content) # Streamingstream = client.chat.completions.create( model=MODEL, messages=[{"role": "user", "content": "Explain speculative decoding in two sentences"}], temperature=1.0, top_p=0.95, stream=True, extra_body={"skip_special_tokens": False},)for chunk in stream: print(chunk.choices[0].delta.content or "", end="", flush=True)
When conversational_mode is enabled, Nemotron-3.5-Bolt responds concisely with minimal markdown formatting—ideal for low-latency chat, voice assistants, and spoken interaction pipelines. It can be combined with enable_thinking: True/False as well as tool calling:
response = client.chat.completions.create( model="bolt", messages=[{"role": "user", "content": "What is the capital of Japan?"}], temperature=1.0, top_p=0.95, extra_body={ "skip_special_tokens": False, "chat_template_kwargs": {"conversational_mode": True}, },)print(response.choices[0].message.content)
tools = [{ "type": "function", "function": { "name": "get_weather", "description": "Get the current weather for a city", "parameters": { "type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"], }, },}] response = client.chat.completions.create( model="bolt", messages=[{"role": "user", "content": "What's the weather in Santa Clara?"}], tools=tools, temperature=1.0, top_p=0.95, extra_body={ "skip_special_tokens": False, "chat_template_kwargs": {"enable_thinking": True}, },)print(response.choices[0].message.tool_calls)
Because the chat template requires a tool message immediately after an assistant turn with tool_calls, long-running background tools can immediately return a "pending" acknowledgement and deliver the completed payload in a subsequent user turn prefixed with TOOL UPDATE::
// 1. Assistant invokes tool{ "role": "assistant", "tool_calls": [{ "id": "call_1", "type": "function", "function": { "name": "build_report", "arguments": "{\"_task_id\":\"task_1\",\"scope\":\"weekly\"}" } }]} // 2. Immediate acknowledgement (allows conversation to continue){ "role": "tool", "tool_call_id": "call_1", "content": "{\"status\":\"pending\",\"task_id\":\"task_1\",\"function\":\"build_report\"}"} // 3. Later completion delivered as a user message{ "role": "user", "content": "TOOL UPDATE: {\"status\":\"completed\",\"task_id\":\"task_1\",\"function\":\"build_report\",\"response\":\"Report is ready.\"}"}
In thinking-enabled mode, you can cap the number of reasoning tokens using NVIDIA's two-step continuation pattern (preserve_thinking=True + continue_final_message=True):
import copyimport requests class BoltThinkingBudgetClient: def __init__(self, base_url, api_key="EMPTY"): self.base_url = base_url.rstrip("/") def _post(self, payload): response = requests.post( f"{self.base_url}/chat/completions", json=payload, timeout=300, ) response.raise_for_status() return response.json() def chat_completion( self, model, messages, max_thinking_budget=512, max_tokens=4096, tools=None, temperature=1.0, top_p=0.95, seed=None, ): if max_tokens <= max_thinking_budget: raise ValueError("max_tokens must be larger than max_thinking_budget") original_messages = copy.deepcopy(messages) sampling = {"temperature": temperature, "top_p": top_p} if seed is not None: sampling["seed"] = seed first_payload = { "model": model, "skip_special_tokens": False, "messages": original_messages, "max_tokens": max_thinking_budget, "chat_template_kwargs": {"enable_thinking": True}, **sampling, } if tools is not None: first_payload["tools"] = tools first_payload["tool_choice"] = "auto" first = self._post(first_payload) choice = first["choices"][0] message = choice["message"] reasoning = message.get("reasoning_content") or message.get("reasoning") or "" partial_answer = message.get("content") or "" tool_calls = message.get("tool_calls") or [] if choice.get("finish_reason") == "stop" and (partial_answer or tool_calls): return { "reasoning_content": reasoning, "content": partial_answer, "tool_calls": tool_calls, "finish_reason": choice.get("finish_reason"), "budget_forced": False, } used_tokens = first.get("usage", {}).get("completion_tokens", max_thinking_budget) remaining_tokens = max_tokens - used_tokens if remaining_tokens <= 0: raise ValueError("Increase max_tokens or lower max_thinking_budget") continuation_messages = copy.deepcopy(original_messages) continuation_messages.append( { "role": "assistant", "reasoning_content": reasoning, "content": partial_answer, } ) second_payload = { "model": model, "skip_special_tokens": False, "messages": continuation_messages, "max_tokens": remaining_tokens, "add_generation_prompt": False, "continue_final_message": True, "chat_template_kwargs": { "enable_thinking": True, "preserve_thinking": True, }, **sampling, } if tools is not None: second_payload["tools"] = tools second_payload["tool_choice"] = "auto" second = self._post(second_payload) second_choice = second["choices"][0] second_message = second_choice["message"] return { "reasoning_content": reasoning, "content": partial_answer + (second_message.get("content") or ""), "tool_calls": second_message.get("tool_calls") or [], "finish_reason": second_choice.get("finish_reason"), "budget_forced": True, }
- Structured System Prompts: Place persistent role definitions, behavioral rules, and output constraints in the
system prompt using clear section headings (ROLE, IDENTITY, INPUT FORMAT, TASK RULES).
- Explicit Length & Action Constraints: Use concrete constraints (e.g., "limit responses to 1–2 sentences") rather than vague instructions ("be brief"), and phrase multi-step workflows as explicit conditions and actions ("After crafting succeeds, call the delivery tool").
- Conveying Environment State (
# CURRENT STATE): Mark operational state updates with # CURRENT STATE in user or system messages in strict chronological order (before a user turn that depends on the state, or immediately after a tool execution response that mutates state).
License
Published under the Open Model Derivative License (OpenMDW-1.1), incorporating permissions and attribution from the upstream NVIDIA Open Model License Agreement and Google Gemma Apache 2.0 license.