Run it
vllm serve turintech/Nemotron-3.5-Bolt-E4B-Artemis-W4A16 \ --dtype bfloat16 --max-model-len 4096 --language-model-only \ --enable-auto-tool-choice --tool-call-parser gemma4
Tested with vLLM 0.32.0rc1 (a development build). Every
result below was measured with thinking off; contexts longer than 4,096 tokens were not tested. On a 16 GB GPU, add
--gpu-memory-utilization 0.92 --max-num-batched-tokens 2048
, and lower the first value if the GPU also drives a display; this is expected to
work, not tested on a 16 GB card. For images and audio, drop
--language-model-only; audio also needs
pip install "vllm[audio]".
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{ "model": "turintech/Nemotron-3.5-Bolt-E4B-Artemis-W4A16", "messages": [{"role": "user", "content": "In two sentences, why is the sky blue?"}], "chat_template_kwargs": {"enable_thinking": false}, "max_tokens": 256}'
The reply is in choices[0].message.content. Tool calls use the same endpoint with a tools list and come back in
choices[0].message.tool_calls.
Results

Table with columns: File, Size, KL chat (lower is closer), Tool calls, uncalibrated cases, GSM8K, first 250, Generation, one request (tokens/s)| File | Size | KL chat (lower is closer) | Tool calls, uncalibrated cases | GSM8K, first 250 | Generation, one request (tokens/s) |
|---|
| TurinTech W4A16 | 11.24 GB | 0.080 | 92.5% | 92.0% | 205 |
| TurinTech FP8 | 12.69 GB | 0.010 | 92.2% | 92.0% | 183 |
| BF16 original | 15.99 GB | reference | 92.2% | 91.6% | 139 |
Bold marks the best in each column. Tool calls leave out the cases used in calibration. One NVIDIA RTX 5090, vLLM 0.32.0rc1, thinking off. Full results, memory and method.
Files
Table with columns: File, Size, Use it for| File | Size | Use it for |
|---|
attn.safetensors | 1.17 GB | Unchanged tensors from NVIDIA's checkpoint, repackaged |
core.safetensors | 8.10 GB | Unchanged tensors from NVIDIA's checkpoint, repackaged |
mlp.safetensors | 1.86 GB | The language model's MLP weights, quantized |
model-frozen-shared-kv.safetensors | 110 MB | Redistributed unchanged from NVIDIA's checkpoint |
chat_template.jinja and 6 more | 32 MB | Configuration, tokenizer, chat template and weight index, which vLLM reads |
bias.md and 6 more | 30 MB | The original publisher's model-card documents, media and handler, unchanged |
LICENSE and 1 more | 5 kB | Licence and notice |
The checksums section lists every file. vllm serve turintech/Nemotron-3.5-Bolt-E4B-Artemis-W4A16 downloads the 11 weight and configuration files, 11.28 GB in all.
Pick this model
- Good for serving with vLLM in 11.24 GB of weights, with 235,490 tokens of KV cache on a 32 GB RTX 5090 and 205 tokens per second one request at a time, with simple tool calls and GSM8K level with the original.
- Pick another file when closeness to the original matters: its KL is 6.0 to 7.8 times our FP8 checkpoint's. Multi-turn tool calls and contexts longer than 4,096 tokens were not tested, and images and audio were only spot-checked.
How it was made
Nemotron 3.5 Bolt E4B is NVIDIA's small chat model for text, images and audio, post-trained by NVIDIA from Google's
Gemma 4 E4B. This checkpoint comes from TurinTech's work with Artemis,
starting from NVIDIA's checkpoint
(nvidia/NVIDIA-Nemotron-3.5-Bolt-E4B-BF16, revision
49843ee). Only the language model's MLP changed: its weights are 4-bit integers with group size 32 and its activations stay BF16 (compressed-tensors, GPTQ with llm-compressor). Attention, the embeddings, the
per-layer gates and projections and the vision and audio towers are NVIDIA's, unchanged. The 4-bit weights were calibrated on public text: WikiText-2 train, OpenAssistant train conversations, the CPython standard library and Berkeley Function Calling Leaderboard items. NVIDIA's
tokenizer, chat template, model-card documents and media are included unchanged.
How it was tested
Every number on this card was measured with vLLM 0.32.0rc1 on one NVIDIA RTX 5090, greedy decoding and thinking off,
against NVIDIA's BF16 original run the same way. Closeness is the KL divergence from the original on wiki, chat and
code text, computed in Hugging Face Transformers. Tasks are BFCL simple tool calls and the first 250 GSM8K
problems. Images and audio were only
spot-checked. The results page has every table, the method and memory.
Table with columns: File, Bytes, sha256| File | Bytes | sha256 |
|---|
LICENSE | 2,619 | 2ab44b68365473c112f5092211a38f231cb23e50de68b75a13369adbd76a74df |
NOTICE | 2,011 | ece00cb9fc7270473cf62e4fe4569058202cffb058e454e3a86476a4b4a6713c |
attn.safetensors | 1,174,422,384 | d13c9218f93f63b5cfc1416c51b6382fccca5ee4648fc7527893071cdbcdb69b |
bias.md | 3,254 | bbe310672de023a587ba92e5c1d5da2edc22955f8822b2f3ffe00afca2dd8de1 |
bolt-smart-and-fast.png | 444,073 | 244b8a0f465c13fbfa81c8046b0bd4471d2202c275ea950c4845374788852145 |
chat_template.jinja | 20,161 | 35b7d0641b6fb4ee41c3bc623e85bdb7d583c40f6ae9f24514ae58a9d57119f3 |
config.json | 5,875 | 781478db0dab8c27f86c3f9f17d2cd0231104787051b5c7c2c215d0e8006b81d |
core.safetensors | 8,102,007,484 | 899fa208df63ab0058dab6dc75f3489967bf1535cd211423a151d2c1f762e88a |
explainability.md | 2,674 | 7671d31882b60bb4eb385e324717688941fb6d1ed99359af7b1603c98018bac5 |
generation_config.json | 208 | d4226bbe3117d2d253ba4609720ba82c6c4ce4627a9a6ae05387c78983ac03de |
handler.py | 7,165 | 48adddae3baf5adfb2cca8412df3c312b9ddc4550146708a53ac77945d4d66be |
mlp.safetensors | 1,857,995,640 | ea4bc40a06bd8ec56242b41c9a2321cdbe37faf5861b1ba0b13d8e075195436a |
model-frozen-shared-kv.safetensors | 110,118,048 | 32aaa38402af5a51c6d6b27aeb8a26d3e6964e5590330f0e7b2f546608da4023 |
model.safetensors.index.json | 201,480 | 9f308e398faa2d7313de10d1f3e85e2f11de7dfa239ae24ae463ad04d4759303 |
privacy.md | 1,671 | c048eb257c10d128d35a8a90d8c98e7684e2019d9e110f7db1543cf88f1ce63a |
processor_config.json | 1,689 | 32bdf45d2ad4cc29a0822ddd157a182de76644f0419a6228d151495256e9813c |
reachy-bolt.gif | 29,796,429 | 446ab42f5a52e2d9d2bb09a76efa4439d96929090eaab0f3d2fce4f9e330cd75 |
safety.md | 2,292 | 50df88aa4e1f25c3504522ed11738075f2d97150c15a89508a1eb6912aec22f0 |
tokenizer.json | 32,169,626 | cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f |
tokenizer_config.json | 23,725 | b3da1b7227c5d35fd905ad13200a08be63cb346b2d1bd4f6d58878c9783d5de1 |
Thanks and feedback
Thanks to the vLLM and llm-compressor projects and to the
open-source community, for pushing the limits of serving models efficiently. Their work was our foundation
throughout, and this checkpoint builds on theirs.
This checkpoint is new for us, and feedback is welcome: whether it runs well on a given GPU, anything that did not
work, or a benchmark that should be measured differently. Please open a discussion on this page.
Artemis is TurinTech's optimisation platform: teams use it to make their own models smaller and faster, or to optimise their code.
All our models · LinkedIn · Questions and requests
OpenMDW-1.1, as released by NVIDIA (licence); see LICENSE and NOTICE. An independent TurinTech release, not affiliated with or endorsed by NVIDIA. When serving this checkpoint to others, for example through an API, an inference provider or OpenRouter, we'd appreciate a mention of TurinTech (turintech.ai) as the source of the quantization.
These files keep improving.