GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
RedHatAI
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
RedHatAI
Model Tree
nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16
Input Modalities
Output Modalities
Supported Functionality
This is an FP8 Dynamic variant of the nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16 checkpoint, prepared for vLLM inference.
The following sections are covered below:
LLM Compressor model-free PTQ applies FP8 Dynamic quantization to selected linear operators in the backbone and MTP modules. Quantized weights use per-channel FP8, and activations use dynamic per-token FP8 scaling. Other tensors retain their original precision. The checkpoint is saved in compressed-tensors format.
Quantized weights use 8 bits instead of 16-bit BF16, reducing storage and memory for those tensors. Overall savings depend on the share of the model that remains in BF16.
To assess the impact of quantization on model accuracy, particularly in reasoning-intensive tasks and agentic workflows, we evaluate both the BF16 and FP8 models across a broad suite of representative benchmarks. To ensure a fair and consistent comparison, both models are evaluated under identical conditions: non-greedy sampling, 4× NVIDIA H200 GPUs, vLLM 0.31.0, and the Inspect AI and Harbor evaluation frameworks. The evaluation results are presented in the figure below.
The FP8 checkpoint consistently preserves model accuracy across all evaluated benchmarks.
The plots demonstrate the performance boost given to the Nemotron 3.5 Super VL (4x H100, temperature 0, max_tokens 1024) when swapped from the original BF16 checkpoint to the FP8 quantized counterpart (this model). Both models were served with MTP using 3 draft tokens, across workloads with prompts from various datasets. The FP8 checkpoint delivers a ~1.2x speedup for a single user, rising to ~1.3-1.4x under concurrent load. As the number of speculative tokens is unchanged, the gain comes purely from faster quantized compute. Each chart plots interactivity, defined as 1000 divided by mean inter-token latency (i.e., the tok/s streaming speed a single user experiences), on the x-axis against aggregate output throughput (tok/s) on the y-axis, with one point per closed-loop concurrency level N. The HumanEval prompts come from OpenAI's HumanEval coding benchmark, while the math_reasoning and RAG prompts come from Spec-Bench, the ACL 2024 speculative decoding benchmark suite.
For further performance gains, the DSpark speculator RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B.dspark can be used in place of MTP. The plots below replicate the above benchmarking setup, but swap to use the DSpark speculator with 8 speculative tokens. See the speculator's model card for serving details.
The following example uses MTP speculative decoding and FlashInfer for the Mamba backend:
text
vllm serve RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8 \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.89 \ --prefix-match-unit 64 \ --block-size 64 \ --enable-mamba-fine-grained-prefix-cache \ --compilation_config.max_cudagraph_capture_size 192 \ --enable-expert-parallel \ --max-num-seqs 128 \ --trust-remote-code \ --max-model-len 114688 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --async-scheduling \ --speculative_config.method mtp \ --speculative_config.num_speculative_tokens 3 \ --mamba-backend flashinfer \ --mamba-ssm-cache-dtype float16 \ --enable-mamba-cache-stochastic-rounding \ --mamba-cache-philox-rounds 5 \ --chat-template-content-format string \ --mamba-cache-mode align \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3The FP8 checkpoint is documented to fit on 2x H100 GPUs, compared with 4 GPUs for the BF16 checkpoint. Hardware needs vary with serving configuration and workload.
This checkpoint was produced with LLM Compressor's model-free PTQ flow using the FP8_DYNAMIC scheme. No calibration dataset is required. The script below uses a BF16 checkpoint as input and saves the quantized checkpoint in compressed-tensors format.
python
"""Apply FP8 Dynamic to the Nemotron model and selected MTP modules.""" import jsonfrom pathlib import Path from compressed_tensors.quantization import QuantizationConfigfrom llmcompressor import model_free_ptq # Set these paths before running the example.MODEL_ID = "/path/to/bf16-checkpoint"SAVE_DIR = "/path/to/fp8-dynamic-checkpoint"DEVICE = "cuda:0" # Match non-MTP modules, then let IGNORE restrict the backbone to eligible linears.TARGETS = [ # Backbone modules, excluding MTP modules. r"re:^(?!.*(?:^|\.)mtp\.).+$", # MTP routed-expert gate, up, and down projections. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.experts\.\d+\." r"(gate_proj|up_proj|down_proj)$", # MTP attention q, k, v, and output projections. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\." r"(q_proj|k_proj|v_proj|o_proj)$", # MTP shared-expert gate, up, and down projections. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.shared_experts\." r"(gate_proj|up_proj|down_proj)$",] IGNORE = [ # Standard exclusions. r"re:.*lm_head$", r"re:.*conv1d.*", r"re:.*backbone\.embeddings$", r"re:.*_latent_proj$", r"re:.*mixer\.gate$", # Vision model and connector. r"re:^vision_model\..*", r"re:^mlp1\..*", # Norms. r"re:.*backbone\.norm_f$", r"re:.*\.norm[12]$", # Backbone shared experts; MTP shared experts remain targeted. r"re:.*(?:^|\.)(?:backbone|model)\.layers\.\d+\.mixer\." r"shared_experts(?:\.|$).*", # MTP routers and prediction head. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.gate$", r"re:.*(?:^|\.)mtp\.layers\.\d+\.eh_proj$",] config = QuantizationConfig( config_groups={"FP8_DYNAMIC": TARGETS}, ignore=IGNORE,) model_free_ptq( model_stub=MODEL_ID, save_directory=SAVE_DIR, config=config, device=DEVICE,) # Keep the top-level and nested quantization metadata in sync for MTP.config_path = Path(SAVE_DIR) / "config.json"saved_config = json.loads(config_path.read_text())if "llm_config" in saved_config: saved_config["llm_config"]["quantization_config"] = saved_config[ "quantization_config" ] config_path.write_text(json.dumps(saved_config, indent=2) + "\n")vllm serve RedHatAI/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-FP8 \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.89 \ --prefix-match-unit 64 \ --block-size 64 \ --enable-mamba-fine-grained-prefix-cache \ --compilation_config.max_cudagraph_capture_size 192 \ --enable-expert-parallel \ --max-num-seqs 128 \ --trust-remote-code \ --max-model-len 114688 \ --max-num-batched-tokens 16384 \ --enable-prefix-caching \ --async-scheduling \ --speculative_config.method mtp \ --speculative_config.num_speculative_tokens 3 \ --mamba-backend flashinfer \ --mamba-ssm-cache-dtype float16 \ --enable-mamba-cache-stochastic-rounding \ --mamba-cache-philox-rounds 5 \ --chat-template-content-format string \ --mamba-cache-mode align \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ --reasoning-parser nemotron_v3"""Apply FP8 Dynamic to the Nemotron model and selected MTP modules.""" import jsonfrom pathlib import Path from compressed_tensors.quantization import QuantizationConfigfrom llmcompressor import model_free_ptq # Set these paths before running the example.MODEL_ID = "/path/to/bf16-checkpoint"SAVE_DIR = "/path/to/fp8-dynamic-checkpoint"DEVICE = "cuda:0" # Match non-MTP modules, then let IGNORE restrict the backbone to eligible linears.TARGETS = [ # Backbone modules, excluding MTP modules. r"re:^(?!.*(?:^|\.)mtp\.).+$", # MTP routed-expert gate, up, and down projections. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.experts\.\d+\." r"(gate_proj|up_proj|down_proj)$", # MTP attention q, k, v, and output projections. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\." r"(q_proj|k_proj|v_proj|o_proj)$", # MTP shared-expert gate, up, and down projections. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.shared_experts\." r"(gate_proj|up_proj|down_proj)$",] IGNORE = [ # Standard exclusions. r"re:.*lm_head$", r"re:.*conv1d.*", r"re:.*backbone\.embeddings$", r"re:.*_latent_proj$", r"re:.*mixer\.gate$", # Vision model and connector. r"re:^vision_model\..*", r"re:^mlp1\..*", # Norms. r"re:.*backbone\.norm_f$", r"re:.*\.norm[12]$", # Backbone shared experts; MTP shared experts remain targeted. r"re:.*(?:^|\.)(?:backbone|model)\.layers\.\d+\.mixer\." r"shared_experts(?:\.|$).*", # MTP routers and prediction head. r"re:.*(?:^|\.)mtp\.layers\.\d+\.mixer\.gate$", r"re:.*(?:^|\.)mtp\.layers\.\d+\.eh_proj$",] config = QuantizationConfig( config_groups={"FP8_DYNAMIC": TARGETS}, ignore=IGNORE,) model_free_ptq( model_stub=MODEL_ID, save_directory=SAVE_DIR, config=config, device=DEVICE,) # Keep the top-level and nested quantization metadata in sync for MTP.config_path = Path(SAVE_DIR) / "config.json"saved_config = json.loads(config_path.read_text())if "llm_config" in saved_config: saved_config["llm_config"]["quantization_config"] = saved_config[ "quantization_config" ] config_path.write_text(json.dumps(saved_config, indent=2) + "\n")