Run it
vllm serve turintech/Nemotron-3.5-Bolt-E4B-Artemis-FP8 \ --dtype bfloat16 --max-model-len 4096 --language-model-only \ --enable-auto-tool-choice --tool-call-parser gemma4
Tested with vLLM 0.32.0rc1 (a development build). Every
result below was measured with thinking off; BFCL-v3 ran at a 65,536-token context and the other checks at 4,096. On a 16 GB GPU, add
--gpu-memory-utilization 0.92 --max-num-batched-tokens 2048
, and lower the first value if the GPU also drives a display; this is expected to
work, not tested on a 16 GB card. For images and audio, drop
--language-model-only; audio also needs
pip install "vllm[audio]".
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "turintech/Nemotron-3.5-Bolt-E4B-Artemis-FP8", "messages": [{"role": "user", "content": "In two sentences, why is the sky blue?"}], "chat_template_kwargs": {"enable_thinking": false}, "max_tokens": 256}'
The reply is in choices[0].message.content. Tool calls use the same endpoint with a tools list and come back in
choices[0].message.tool_calls.
With NVIDIA's DFlash drafter:
vllm serve turintech/Nemotron-3.5-Bolt-E4B-Artemis-FP8 --dtype bfloat16 --max-model-len 8192 --language-model-only \ --enable-auto-tool-choice --tool-call-parser gemma4 \ --speculative-config '{"method":"dflash","model":"nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16-DFlash-NVFP4","num_speculative_tokens":7}'
Results

Table with columns: File, Size, KL chat (lower is closer), Tool calls (of 400), GSM8K (of 250), Generation, one request (tokens/s)| File | Size | KL chat (lower is closer) | Tool calls (of 400) | GSM8K (of 250) | Generation, one request (tokens/s) |
|---|
| TurinTech FP8 | 12.69 GB | 0.010 | 371 | 230 | 183 |
| BF16 original | 15.99 GB | reference | 371 | 229 | 139 |
Bold marks the best in each column. One NVIDIA RTX 5090, vLLM 0.32.0rc1, thinking off. Full results, memory and method.
Files
Table with columns: File, Size, Use it for| File | Size | Use it for |
|---|
part-00000.safetensors | 1.96 GB | Unchanged tensors from NVIDIA's checkpoint, repackaged |
part-00001.safetensors | 5.64 GB | Unchanged tensors from NVIDIA's checkpoint, repackaged |
part-00002.safetensors | 2.47 GB | Weights, the MLP among them in FP8 |
part-00003.safetensors | 2.62 GB | Weights, the MLP among them in FP8 |
chat_template.jinja and 6 more | 32 MB | Configuration, tokenizer, chat template and weight index, which vLLM reads |
bias.md and 6 more | 30 MB | The original publisher's model-card documents, media and handler, unchanged |
LICENSE and 1 more | 4 kB | Licence and notice |
The checksums section lists every file. vllm serve turintech/Nemotron-3.5-Bolt-E4B-Artemis-FP8 downloads the 11 weight and configuration files, 12.73 GB in all.
Pick this model
- Good for serving with vLLM in 12.69 GB of weights, with 37% more KV cache than the original and generation 32 to 43% faster, at the same BFCL and GSM8K scores within one question.
- Pick another file when the smallest weights or the fastest generation matter more than closeness: the W4A16 and NVFP4 checkpoints are smaller and generate faster, further from the original. Images and audio were only spot-checked, and a 16 GB GPU was not tested.
How it was made
Nemotron 3.5 Bolt E4B is NVIDIA's small chat model for text, images and audio, post-trained by NVIDIA from Google's
Gemma 4 E4B. This checkpoint comes from TurinTech's work with Artemis,
starting from NVIDIA's checkpoint
(nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, revision
49843ee). Only the language model's MLP changed: its weights are FP8 E4M3 with one scale per output channel and its activations stay BF16 (compressed-tensors, weights only). Attention, the embeddings, the
per-layer gates and projections and the vision and audio towers are NVIDIA's, unchanged. It used no calibration data. NVIDIA's
tokenizer, chat template, model-card documents and media are included unchanged.
How it was tested
Every number on this card was measured with vLLM 0.32.0rc1 on one NVIDIA RTX 5090, greedy decoding and thinking off,
against NVIDIA's BF16 original run the same way. Closeness is the KL divergence from the original on wiki, chat and
code text, computed in Hugging Face Transformers. Tasks are BFCL simple tool calls and the first 250 GSM8K
problems; the full BFCL-v3 and all 1,319 GSM8K problems were also run. Images and audio were only
spot-checked. The results page has every table, the method and memory.
Table with columns: File, Bytes, sha256| File | Bytes | sha256 |
|---|
LICENSE | 2,619 | 2ab44b68365473c112f5092211a38f231cb23e50de68b75a13369adbd76a74df |
NOTICE | 1,676 | 9c904e6b0c7b22f107d895c0f6c9b58e4e54c72a33e20d12a9219e6c4d9e7eba |
bias.md | 3,254 | bbe310672de023a587ba92e5c1d5da2edc22955f8822b2f3ffe00afca2dd8de1 |
bolt-smart-and-fast.png | 444,073 | 244b8a0f465c13fbfa81c8046b0bd4471d2202c275ea950c4845374788852145 |
chat_template.jinja | 20,161 | 35b7d0641b6fb4ee41c3bc623e85bdb7d583c40f6ae9f24514ae58a9d57119f3 |
config.json | 44,847 | 5c57a219998d6a9771c7e82ea9b2e789f9ae0319e98814395806760a04559542 |
explainability.md | 2,674 | 7671d31882b60bb4eb385e324717688941fb6d1ed99359af7b1603c98018bac5 |
generation_config.json | 208 | d4226bbe3117d2d253ba4609720ba82c6c4ce4627a9a6ae05387c78983ac03de |
handler.py | 7,165 | 48adddae3baf5adfb2cca8412df3c312b9ddc4550146708a53ac77945d4d66be |
model.safetensors.index.json | 207,744 | 7913f100e23227ae86fe2d0a4a3b19686c0ec78124773e441b7174ccd9f6b963 |
part-00000.safetensors | 1,963,720,280 | cfce28a3227026275e8d19c7d9358b461e3945475a9ac8ff9d18183140a30425 |
part-00001.safetensors | 5,637,144,744 | f2c2cea5c4855c25f9004e8db7fdaf04149e75fc8eb11a1b1466f3f44c78db0e |
part-00002.safetensors | 2,468,051,980 | 83150153e4ec963436e7df0c2cf92b32bdc73abea603b46a6572c05b28c0a22b |
part-00003.safetensors | 2,624,548,008 | 6b1909d2c589f80d5d927568287c835fa6bc4547af5851f9abac966387d5f96b |
privacy.md | 1,671 | c048eb257c10d128d35a8a90d8c98e7684e2019d9e110f7db1543cf88f1ce63a |
processor_config.json | 1,689 | 32bdf45d2ad4cc29a0822ddd157a182de76644f0419a6228d151495256e9813c |
reachy-bolt.gif | 29,796,429 | 446ab42f5a52e2d9d2bb09a76efa4439d96929090eaab0f3d2fce4f9e330cd75 |
safety.md | 2,292 | 50df88aa4e1f25c3504522ed11738075f2d97150c15a89508a1eb6912aec22f0 |
tokenizer.json | 32,169,626 | cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f |
tokenizer_config.json | 23,725 | b3da1b7227c5d35fd905ad13200a08be63cb346b2d1bd4f6d58878c9783d5de1 |
Thanks and feedback
Thanks to the vLLM and llm-compressor projects and to the
open-source community, for pushing the limits of serving models efficiently. Their work was our foundation
throughout, and this checkpoint builds on theirs.
This checkpoint is new for us, and feedback is welcome: whether it runs well on a given GPU, anything that did not
work, or a benchmark that should be measured differently. Please open a discussion on this page.
Artemis is TurinTech's optimisation platform: teams use it to make their own models smaller and faster, or to optimise their code.
All our models · LinkedIn · Questions and requests
OpenMDW-1.1, as released by NVIDIA (licence); see LICENSE and NOTICE. An independent TurinTech release, not affiliated with or endorsed by NVIDIA. When serving this checkpoint to others, for example through an API, an inference provider or OpenRouter, we'd appreciate a mention of TurinTech (turintech.ai) as the source of the quantization.
These files keep improving.