GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
88plug
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
88plug
Model Tree
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16
Input Modalities
Output Modalities
Supported Functionality
These weights are compressed-tensors (pack-quantized / int-quantized).
| Runtime | Supported |
|---|---|
| vLLM ≥ 0.21 | Yes — preferred (auto-detect CT; no --quantization flag) |
transformers + compressed-tensors | Yes for many text models; multimodal may need custom code |
| Text Generation Inference (TGI) | Not supported for these CT packs |
| Hugging Face Inference Widget | Often fails — use vLLM locally instead |
bash
# Preferredvllm serve 88plug/<ModelName> --trust-remote-codeDo not deploy via TGI “text-generation-inference” paths — that backend does not load our CT format and produces opaque worker/load errors.
INT4 post-training quantization of nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 — NVIDIA's hybrid Mamba-Transformer MoE 30B model with 3B active parameters. ~15 GB on disk. Runs on a single RTX 3090 or A40.
| Property | Value |
|---|---|
| Base model | nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 |
| Release tier | Gold (AutoRound iters=200) |
| Quant method | AutoRound W4A16 iters=200 attention + datafree RTN MoE (Mamba/Triton, device-forced) |
| FLAC status | Not measured (T+7d milestone) |
| Architecture | Hybrid Mamba2 SSM + Transformer attention, MoE |
| Active parameters | 3B (MoE sparse activation) |
| Quant format | compressed-tensors (native vLLM) |
| Targets | All Linear layers (see exclusions below) |
| Disk size | ~15 GB |
| Min GPU | 1× RTX 3090 24GB or RTX A40 48GB |
This recipe targets every Linear module in the model except the following, which remain in BF16:
| Excluded pattern | Layer type | Reason |
|---|---|---|
lm_head | Output projection | Vocabulary logit precision |
.*embed_tokens$ | Embedding table | Input token representation |
.*norm$ | Layer norms | Normalization stability |
.*mixer\.in_proj$ | Mamba2 SSM input projection | SSM dynamics degrade severely under INT4 |
.*mixer\.out_proj$ | Mamba2 SSM output projection | SSM dynamics degrade severely under INT4 |
.*\.gate$ | MoE router gate | Routing-critical, small matrix |
.*moe_gate.* | MoE gate (alternate naming) | Routing-critical |
What is quantized to W4A16: All transformer attention projections (Q/K/V/O) and all MoE expert Linear layers (up/down/gate projections inside each expert), excluding the router gate. Mamba2 SSM layers (in_proj, out_proj inside mixer) are fully excluded.
Mamba2's selective-scan operation is highly sensitive to weight precision — INT4 induces state-space divergence that accumulates across sequence length. Additionally, mamba_ssm's selective scan is not torch.fx-traceable; the pipeline uses tracing_ignore=["mixer"] and sequential_targets=["Linear"] to route around this at calibration time.
| Configuration | BF16 | W8A16 (88plug) | This Quant (W4A16) |
|---|---|---|---|
| Weights (disk/VRAM) | ~60 GB | ~30 GB | ~15 GB |
| KV cache @ 32k ctx (fp8) | ~2.0 GB | ~2.0 GB | ~2.0 GB |
| Total @ 32k ctx | ~62 GB | ~32 GB | ~17 GB |
| Minimum GPU | 2× A100 40GB | 1× A100 40GB | 1× RTX 3090 24GB |
The 30B parameter count is the total; only ~3B are active per forward pass (MoE sparse activation), but all expert weights must reside in VRAM. Mamba2 SSM layers (excluded from quantization) contribute a small fraction of the total weight footprint.
Tested with vLLM v0.21.0 (vllm/vllm-openai:v0.21.0-cu129-ubuntu2404). Weights are in compressed-tensors format — vLLM detects and loads quantization automatically. No --quantization flag needed.
bash
docker run --gpus device=0 -p 8080:8080 \ vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \ 88plug/Nemotron-3-Nano-30B-A3B-W4A16 \ --kv-cache-dtype fp8 \ --max-model-len 32768 \ --gpu-memory-utilization 0.92Weights are in compressed-tensors format — no --quantization flag needed.
python
from openai import OpenAI client = OpenAI(base_url="http://localhost:8080/v1", api_key="none") response = client.chat.completions.create( model="88plug/Nemotron-3-Nano-30B-A3B-W4A16", messages=[{"role": "user", "content": "Explain mixture-of-experts routing in one paragraph."}], max_tokens=512, temperature=0.6,)print(response.choices[0].message.content)SGLang v0.5.8 has Mamba dual-pool support. Verify nemotron_h (hybrid Mamba-Transformer) architecture is recognized before production use. SGLang does not natively consume compressed-tensors; run against the BF16 base for baseline comparisons.
bash
docker run --gpus device=0 -p 30000:30000 \ lmsysorg/sglang:v0.5.8-cu129 python -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 \ --tp 1 \ --mem-fraction-static 0.85 \ --port 30000Text-only model — single GGUF, no mmproj needed. Verify nemotron_h architecture support in your llama.cpp build before use.
bash
python convert_hf_to_gguf.py nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 \ --outfile Nemotron-3-Nano-30B-BF16.gguf llama-quantize Nemotron-3-Nano-30B-BF16.gguf Nemotron-3-Nano-30B-Q8_0.gguf Q8_0llama-quantize --imatrix calibration_datav3.txt \ Nemotron-3-Nano-30B-BF16.gguf Nemotron-3-Nano-30B-IQ4_XS.gguf IQ4_XS llama-server \ --model Nemotron-3-Nano-30B-Q8_0.gguf \ --n-gpu-layers 999 \ --ctx-size 32768 \ --port 8081| Metric | W4A16 Target |
|---|---|
| KL divergence vs BF16 | < 0.014 |
| MMLU recovery | ≥ 99% |
| RULER@128k recovery | ≥ 97% |
| Model | Source | Format | Compare angle |
|---|---|---|---|
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 | official | BF16 | Quality ceiling |
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-FP8 | official | FP8 | Official quant (requires FP8-capable GPU) |
nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 | official | NVFP4 | Official W4-class (H100/Blackwell only) |
88plug/Nemotron-3-Nano-30B-A3B-W8A16 | 88plug | W8A16 compressed-tensors | Higher-precision 88plug quant |
Release highlight: NVFP4 requires H100/Blackwell tensor cores. This W4A16 compressed-tensors quant runs on any CUDA GPU with sufficient VRAM (RTX 3090+), using vLLM's Marlin kernel — no special hardware required.
Results pending.
| Engine | Format | Batch | ctx | tok/s | TTFT p50 | TTFT p99 | VRAM |
|---|---|---|---|---|---|---|---|
| vLLM v0.21.0 | W4A16 compressed-tensors | 1 | 32k | — | — | — | — |
| vLLM v0.21.0 | W4A16 compressed-tensors | 8 | 32k | — | — | — | — |
| vLLM v0.21.0 | W4A16 compressed-tensors | 1 | 128k | — | — | — | — |
| SGLang v0.5.8 | BF16 (baseline) | 1 | 32k | — | — | — | — |
| llama.cpp b9297 | Q8_0 GGUF | 1 | 32k | — | — | — | — |
| llama.cpp b9297 | IQ4_XS GGUF | 1 | 32k | — | — | — | — |
Hardware: A6000 48 GB, CUDA 12.9, driver 570.
| Parameter | Value |
|---|---|
| Calibration samples | 1024 (512 UltraChat-200k + 512 wikitext-103-raw-v1) |
| Max sequence length | 2048 tokens |
| UltraChat split | train_sft[:512] |
| Wikitext filter | len(text) > 512, shuffled seed=42 |
| Dataset shuffle seed | 42 |
All MoE expert Linear layers (up/down/gate-proj inside each expert) are quantized to W4A16. The MoE router gate matrices (.*\.gate$, .*moe_gate.*) are excluded and kept in BF16 — routing decisions are precision-sensitive and the matrices are small enough that BF16 adds negligible VRAM overhead.
mamba_ssm selective scan is not torch.fx-traceable. The pipeline uses two belt-and-suspenders mechanisms:
tracing_ignore=["mixer"] — prevents the tracer from descending into SSM modules.sequential_targets=["Linear"] — processes each Linear individually, bypassing block-level tracing entirely.Calibration requires real mamba_ssm CUDA kernels (installed automatically if absent).
Calibration ran on an RTX A6000 (48 GB). The model is loaded on CPU (device_map=None) with low_gpu_mem_usage=True to cache activations on CPU. This is required to avoid OOM even on 48 GB with 1024 × 2048 activation cache for a ~30B model.
bibtex
@misc{nemotronnanoreport, title = {Nemotron-Nano: A Family of Small Language Models}, author = {NVIDIA}, year = {2025}, url = {https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16}}88plug AI Lab ships FLAC-target compressed-tensors quantizations — AutoRound iters=200, native vLLM v0.21.0+, no extra flags.
This release: Gold tier — full AutoRound calibration (1024 samples, UltraChat + WikiText-103). Targets ≥99% MMLU recovery.
All weights use compressed-tensors format. vLLM reads quantization_config automatically.
Browse all releases → huggingface.co/88plug
# Preferredvllm serve 88plug/<ModelName> --trust-remote-codedocker run --gpus device=0 -p 8080:8080 \ vllm/vllm-openai:v0.21.0-cu129-ubuntu2404 vllm serve \ 88plug/Nemotron-3-Nano-30B-A3B-W4A16 \ --kv-cache-dtype fp8 \ --max-model-len 32768 \ --gpu-memory-utilization 0.92from openai import OpenAI client = OpenAI(base_url="http://localhost:8080/v1", api_key="none") response = client.chat.completions.create( model="88plug/Nemotron-3-Nano-30B-A3B-W4A16", messages=[{"role": "user", "content": "Explain mixture-of-experts routing in one paragraph."}], max_tokens=512, temperature=0.6,)print(response.choices[0].message.content)docker run --gpus device=0 -p 30000:30000 \ lmsysorg/sglang:v0.5.8-cu129 python -m sglang.launch_server \ --model-path nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 \ --tp 1 \ --mem-fraction-static 0.85 \ --port 30000python convert_hf_to_gguf.py nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 \ --outfile Nemotron-3-Nano-30B-BF16.gguf llama-quantize Nemotron-3-Nano-30B-BF16.gguf Nemotron-3-Nano-30B-Q8_0.gguf Q8_0llama-quantize --imatrix calibration_datav3.txt \ Nemotron-3-Nano-30B-BF16.gguf Nemotron-3-Nano-30B-IQ4_XS.gguf IQ4_XS llama-server \ --model Nemotron-3-Nano-30B-Q8_0.gguf \ --n-gpu-layers 999 \ --ctx-size 32768 \ --port 8081@misc{nemotronnanoreport, title = {Nemotron-Nano: A Family of Small Language Models}, author = {NVIDIA}, year = {2025}, url = {https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16}}