GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
cloudnathan5
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
cloudnathan5
Model Tree
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16
Input Modalities
Output Modalities
Supported Functionality
| Component | Weights | Activations |
|---|---|---|
Routed experts (mixer.experts.*.up_proj / down_proj) | NVFP4 (E2M1, group size 16, FP8 E4M3 block scales + FP32 global scale) | BF16 (W4A16) |
Shared experts (mixer.shared_experts.up_proj / down_proj) | NVFP4 (E2M1, group size 16, FP8 E4M3 block scales + FP32 global scale) | BF16 (W4A16) |
Mamba-2 in_proj / out_proj | FP8 E4M3, per-tensor | FP8 E4M3, per-tensor static (calibrated) |
Attention q_proj / k_proj / v_proj / o_proj | BF16 | BF16 |
| KV cache | — | FP8 E4M3 |
MoE latent projections (fc1_latent_proj / fc2_latent_proj) | BF16 | BF16 |
MoE router (gate) | BF16 | BF16 |
Mamba-2 conv1d, SSM parameters, norms | BF16 | BF16 |
Embeddings, lm_head | BF16 | BF16 |
Vision encoder (C-RADIO v4-H) and projector (mlp1) | BF16 | BF16 |
MIXED_PRECISION checkpoint (hf_quant_config.json and quantization_config in config.json), ~47.8 GBsh
wget https://huggingface.co/cloudnathan5/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-NVFP4/resolve/main/ultra_v3_reasoning_parser.py docker run --gpus all --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v "$PWD/ultra_v3_reasoning_parser.py:/parsers/ultra_v3_reasoning_parser.py" \ -e VLLM_USE_V2_MODEL_RUNNER=1 \ vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989 \ cloudnathan5/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-NVFP4 \ --trust-remote-code \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --enable-chunked-prefill \ --attention-backend FLASHINFER \ --mamba-backend flashinfer \ --mamba-cache-mode align \ --kv-cache-dtype fp8 \ --limit-mm-per-prompt '{"video":1,"image":1}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin /parsers/ultra_v3_reasoning_parser.py \ --reasoning-parser ultra_v3wget https://huggingface.co/cloudnathan5/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-NVFP4/resolve/main/ultra_v3_reasoning_parser.py docker run --gpus all --ipc=host -p 8000:8000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ -v "$PWD/ultra_v3_reasoning_parser.py:/parsers/ultra_v3_reasoning_parser.py" \ -e VLLM_USE_V2_MODEL_RUNNER=1 \ vllm/vllm-openai:nightly-77154ca6116ae34cbfe314beef1016bde9b63989 \ cloudnathan5/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-NVFP4 \ --trust-remote-code \ --tensor-parallel-size 1 \ --max-model-len 262144 \ --enable-chunked-prefill \ --attention-backend FLASHINFER \ --mamba-backend flashinfer \ --mamba-cache-mode align \ --kv-cache-dtype fp8 \ --limit-mm-per-prompt '{"video":1,"image":1}' \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser-plugin /parsers/ultra_v3_reasoning_parser.py \ --reasoning-parser ultra_v3