soyrsoyr

NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-MTP

README

License: openmdw-1.1

Compression Recipe

python
from transformers import AutoModelForCausalLM

from llmcompressor import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier
from llcompressor.utils import load_context

model_id = "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"

recipe = QuantizationModifier(
    targets="Linear",
    scheme="NVFP4",
    ignore=[
        r"re:.*conv1d.*",
        r"backbone\.embeddings",
        r"re:.*\_latent_proj.*",
        r"re:.*mixer\.gate\..*",
        "backbone.norm_f",
        "lm_head",
    ],
)

with load_context(AutoModelForCausalLM):
    model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")

oneshot(
    model=model,
    recipe=recipe,
    dataset="perfectblend",
    splits="train[:8]",
    output_dir="/workspace/models/Nemotron-3.5-Lightning-30B-NVFP4-MTP",
    max_seq_length=256,
    num_calibration_samples=8,
    batch_size=1,
    moe_calibrate_all_experts=True,
    mtp_scheme="NVFP4",
)

Transformers does not construct Nemotron's MTP modules when loading the causal LM. LLM Compressor therefore reads those tensors from the source checkpoint during oneshot post-processing, packs eligible MTP weights, writes a dedicated MTP shard, and updates the safetensors index and compressed-tensors config. Static MTP activation scales cannot be calibrated without executable MTP modules, so the MTP group is weight-only NVFP4. The loaded backbone retains its calibrated W4A4 configuration.

Validation

All 25,051 indexed tensors are present across the backbone and MTP shards, with no missing or unindexed tensors.
All 796 MTP tensors are isolated in model_mtp.safetensors; no backbone tensors are present in that shard.
Both source MTP blocks are preserved: mtp.layers.0 is full attention and mtp.layers.1 is MoE.
All 263 eligible MTP weights have NVFP4 packed values, block scales, and global scales. Norms and the MoE router remain BF16.
All 6,004 quantized backbone targets have calibrated input activation scales; the MTP group has none by design.
The NemotronH MTP serving path passed an H100 FP8 Dynamic proxy test with one speculative token. Native NVFP4 serving for this full checkpoint is pending the Blackwell smoke test.

vLLM

bash
vllm serve soyrsoyr/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-MTP \
  --tensor-parallel-size 2 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

soyrsoyr

Model Tree

Base

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

Quantized

this model

Input Modalities

Text

Output Modalities