GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
urakozz
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
urakozz
Model Tree
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16
Input Modalities
Output Modalities
Supported Functionality
| Setting | Value |
|---|---|
| Quantization method | AutoRound |
| Weight and activation scheme | W4A16 |
| Weight format | Symmetric INT4 |
| Group size | 64 |
| Export format | auto_round:auto_gptq |
| Calibration dataset | nvidia/OpenCodeInstruct (opencode-instruct) |
| Calibration samples | 512 |
| Sequence length | 2,048 tokens |
| Optimization iterations | 200 |
| Source model precision | BF16 |
The calibration run used text-only OpenCodeInstruct data. The vision modules were not quantized.
The exported checkpoint includes custom model code from the source model. Use a Transformers version compatible with the checkpoint and enable remote code only when you trust the model repository:
python
from transformers import AutoProcessor, AutoModelForCausalLM model_id = "urakozz/Nemotron-3.5-Bolt-74B-A9B-W4A16-g64-AutoRound-GPTQ"processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True,)This is a large 74B-parameter model. Runtime memory depends on the inference framework, context length, and concurrency. The checkpoint was created in AutoRound's auto_round:auto_gptq format; confirm that your chosen runtime supports this format and the Nemotron-H Omni architecture before deployment.
The source model and its derivatives are governed by the OpenMDW License Agreement, version 1.1. See the accompanying LICENSE file. Please also review the upstream model card for model-specific terms and information.
from transformers import AutoProcessor, AutoModelForCausalLM model_id = "urakozz/Nemotron-3.5-Bolt-74B-A9B-W4A16-g64-AutoRound-GPTQ"processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)model = AutoModelForCausalLM.from_pretrained( model_id, torch_dtype="auto", device_map="auto", trust_remote_code=True,)