GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
GotoAI-Inc
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
GotoAI-Inc
Model Tree
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Input Modalities
Output Modalities
Supported Functionality
Weights were quantized from bfloat16 to int4, group size 64, symmetric, weight-only
(activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was
used and the model was never loaded — the quantizer operates directly on the safetensors.
Architecture, tokenizer and chat template are the vendor's, unmodified.
6,004 Linear modules were converted, covering 80.0% of the output's bytes:
| component | precision | source | quantized |
|---|---|---|---|
| routed experts (128 per MoE layer × 23 layers) | int4 g64 | 58.75 GB | 15.61 GB |
| Mamba in/out projections + attention (70 modules) | int4 g64 | 2.06 GB | 0.55 GB |
| shared expert (46 modules) | int4 g64 | 0.92 GB | 0.24 GB |
MTP head (mtp.*) | bfloat16 | 2.67 GB | 2.67 GB |
backbone.embeddings + lm_head (untied) | bfloat16 | 1.41 GB | 1.41 GB |
MoE routers (.gate) | bfloat16 / fp32 | 0.02 GB | 0.02 GB |
Mamba conv1d, norms, SSM params | bfloat16 | 0.001 GB | 0.001 GB |
| total | 65.83 GB | 20.49 GB |
Group size 64, not the usual 128. The expert down_proj takes an 1856-wide input, and
128 does not divide it. At the default group size 2,944 expert tensors cannot be
quantized; at 64 nothing is misaligned.
Left at bfloat16:
mtp.* — the multi-token-prediction head (one attention + one moe block,
num_nextn_predict_layers: 1). vLLM loads it through the speculative-decoding path
rather than the main stack. At 2.67 GB it is the largest 16-bit component here, so
there is real headroom for anyone who measures that vLLM accepts a quantized one..gate — the 23 MoE routers plus their fp32 e_score_correction_bias. Routing
decides which experts run at all; 0.02 GB is not worth the risk.backbone.embeddings and lm_head — precision-sensitive, and untied in this model.conv1d — Mamba causal-convolution kernels, shape (6144, 1, 4). Not Linear
layers, and quantizers reject them outright.The architecture is a 52-block hybrid: 23 Mamba-2 blocks, 23 MoE blocks and 6 attention blocks. Only 6 blocks carry a KV cache, which is what makes long context cheap here (see below). Of ~30B total parameters, ~3B are active per token — 6 of 128 routed experts plus one shared expert.
Experts ship as per-expert 2-D weights in the source already
(…mixer.experts.{id}.up_proj, down_proj), so no fused-3-D splitting was involved and
the naming carries straight through as …experts.{id}.up_proj.weight_packed. vLLM's
NemotronH loader builds its expert mapping with ckpt_gate_proj_name="up_proj" and
ckpt_down_proj_name="down_proj", matching this layout.
Requires vLLM >= 0.25.1, where NemotronHForCausalLM and the nemotron_v3 reasoning
parser are both present. No nightly build needed.
bash
vllm serve GotoAI-Inc/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16 \--max-model-len 262144 \--mamba-backend flashinfer \--reasoning-parser nemotron_v3 \--enable-auto-tool-choice --tool-call-parser qwen3_coder
Do not pass --quantization; compressed-tensors is detected from config.json. The int4
W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.
--tool-call-parser qwen3_coder is what the base model card specifies — Nemotron 3.5
emits the same XML tool-call framing as Qwen3-Coder. qwen3_xml is an alias for the same
parser class in current vLLM.--reasoning-parser nemotron_v3 splits thinking into reasoning_content.--mamba-ssm-cache-dtype float16 halves the Mamba state cache if you are tight on
memory; the base card pairs it with
--enable-mamba-cache-stochastic-rounding --mamba-cache-philox-rounds 5.nemotron_h_mtp path — --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'.
Neither path is smoke-tested here.Only 6 of 52 blocks use attention, with 2 KV heads at head_dim 128 — about
6 KB/token, an order of magnitude cheaper than a conventional 30B. The 23 Mamba blocks
hold a fixed-size recurrent state instead, roughly 48 MB per concurrent sequence at the
default fp32 SSM cache (half that at float16), independent of sequence length.
| context | KV cache | + weights |
|---|---|---|
| 32k | ~0.2 GB | ~20.7 GB |
| 128k | ~0.8 GB | ~21.3 GB |
| 256k (native max) | ~1.6 GB | ~22.1 GB |
A 32 GB card is comfortable at the full native context with room for concurrency; 24 GB is
workable at moderate context and low concurrency. max_position_embeddings is 262144 —
the base card's validated 1M-token configurations use
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 --max-model-len 1048576 on 8×H100 or GB200, which is not
what this single-card build is for. This is arithmetic from config.json, not a measured
deployment.
Built with llm-quantizer:
bash
./llmq.py run --profile nemotron-3.5-lightning-30b-a3b
which is equivalent to:
python
# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,# transformers==5.15.1, torch==2.13.0from llmcompressor import model_free_ptqmodel_free_ptq(model_stub="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16",save_directory="NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16",scheme="W4A16",group_size=64,ignore=["re:.*\\.gate$", "re:.*\\.conv1d$", "re:.*mtp.*","lm_head", "re:.*\\.embeddings$", "re:.*\\.norm_f$"],device="cuda:0",)
The source ships as 14 shards of ~5 GB and a job holds one shard at a time, so no re-sharding is needed and the build peaks at a few GB of VRAM — it took about a minute on one consumer GPU.
Two of those ignore patterns are easy to miss. re:.*\.embeddings$ is needed because this
model calls its embedding table backbone.embeddings, not embed_tokens. And
re:.*\.norm_f$ is needed because compressed-tensors auto-skips norms with a literal
module_name.endswith("norm") test, which the final norm — backbone.norm_f, 1-D
(2688,) — misses; without it the run aborts with expected 2D linear weight.
No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Treat the published Nemotron 3.5 Lightning numbers as describing the bfloat16 model, not this one.
Two reasons to be more careful than usual here. The routers stay 16-bit, so expert selection is unchanged, but all 2,944 routed experts are quantized without calibration and rarely-activated experts get no more attention than hot ones. And Mamba blocks carry state across the whole sequence, so projection error has a longer path to accumulate than in a pure attention stack — long-context behaviour is the thing worth checking on your own traffic.
OpenMDW License Agreement, version 1.1, inherited from the base model — the vendor's
LICENSE is included unmodified. OpenMDW is permissive: it grants use without restriction
and places no conditions on model outputs. It does require that any distribution retain a
copy of the agreement and all notices of origin, so NVIDIA's LICENSE and its
accompanying safety.md, bias.md, privacy.md and explainability.md are carried
through here. It also terminates the grant for anyone who brings patent or copyright
litigation over the model. Read it before redistributing a derivative.
"Nemotron" and "NVIDIA" are NVIDIA's marks; this repository is not endorsed by or affiliated with NVIDIA.
backbone.norm_fA_logdt_biasD