GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
useful-quants
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
useful-quants
Model Tree
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Input Modalities
Output Modalities
Supported Functionality
63a200063804e06fdb41d6717e43bc92f67859d2LICENSE.This is a third-party, post-training quantization. useful-quants did not train, fine-tune, align, or otherwise develop Nemotron. The conversion does not change NVIDIA's intended-use, safety, or license terms; see the upstream model card for those terms, language coverage, architecture, reasoning behavior, training/data disclosure, and NVIDIA's source-model benchmark suite.
Context terminology matters: NVIDIA specifies an architectural model context of up to 1M tokens. The 640K recommendation and 744K maximum here are deployment validations for this exact W4A16 artifact on one 24 GB RTX 3090 profile; they are not model-wide limits or guarantees for other hardware/software configurations.
Install a current vLLM release compatible with your CUDA/PyTorch installation, then serve the model:
bash
vllm serve useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16 \ --quantization compressed-tensors \ --max-num-seqs 32On the validated vLLM 0.27.1 stack, automatic dtype selection resolves to BF16 for this checkpoint. --dtype bfloat16 is optional but is used in the reproducible 3090 profile to make that choice explicit. --max-num-seqs 32 is required on this 24 GB hybrid-Mamba deployment: vLLM's generic 256-slot default exceeds the available Mamba cache blocks. This compressed-tensors artifact was validated with vLLM; direct Transformers inference is not documented here.
bash
curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16", "messages": [{"role": "user", "content": "Explain why the sky appears blue."}], "temperature": 1.0, "top_p": 0.95, "max_tokens": 512, "chat_template_kwargs": {"enable_thinking": true} }'python
from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")response = client.chat.completions.create( model="useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16", messages=[{"role": "user", "content": "Explain why the sky appears blue."}], temperature=1.0, top_p=0.95, max_tokens=512, extra_body={"chat_template_kwargs": {"enable_thinking": True}},)print(response.choices[0].message.content)NVIDIA's chat template supports enable_thinking=True for reasoning mode and False for direct-answer mode. The temperature=1.0 and top_p=0.95 example values are NVIDIA's upstream sampling defaults, not settings derived by this quantization.
The reproducible production profile is examples/start_vllm_3090_640k.sh. It uses vLLM 0.27.1, PyTorch 2.13, Triton 3.7.1, compiled CUDA graphs, BF16/FP32 cache behavior, priority scheduling, chunked prefill, 32 sequences, 8,192 batched tokens, and a fixed 4,466,765,988-byte cache reservation:
bash
bash examples/start_vllm_3090_640k.shObserved with that profile: peak VRAM was 24,076 MiB (~98.0%); a 655,344-token near-limit single request plus decode passed; and 32 concurrent resident requests totaling 670,688 prompt tokens passed. The deterministic capacity stress had no OOM, preemption, or recomputation. Explicit cache reservation makes the memory envelope reproducible across restarts while retaining CUDA graphs; VLLM_DISABLE_COMPILE_CACHE=1 is intentionally not used.
The validated boundary sweep on the preceding 3090 stack found 736K and 744K successful. At 744K, near-limit decode and 32-way residency both passed, but there was less than one cache page of nominal slack. At 748K the engine could not initialize: it required 4.44 GiB of cache against 4.43 GiB available. Thus 744K is a demonstrated boundary, not an operational recommendation; 640K is deliberately retained as the production setting.
With --max-num-batched-tokens 16384, estimated cache capacity fell to 563,824 tokens and the server could not initialize at 640K. 8192 is therefore the validated throughput/context compromise on this 24 GB profile.
For long-running mixed workloads, vLLM can interleave short batch work and long agentic contexts with priority scheduling. A high-priority request cannot, however, displace running work solely to obtain a sequence slot when all 32 slots are occupied. Reserve 1–2 ingress slots for high-priority/agent traffic: for example, admit bulk work to at most 30–31 slots.
Current-stack vLLM 0.27.1 measurements on the RTX 3090 profile above, using steady-state decode after warm-up. The full 11-point curve is more useful than a three-point headline:
| Active requests | Aggregate decode throughput |
|---|---|
| 1 | 214.7 tok/s |
| 8 | 817.4 tok/s |
| 16 | 1,099.9 tok/s |
| 32 | 1,462.8 tok/s |
This reader-facing comparison uses the same Nemotron 3.5 Lightning 30B-A3B
checkpoint lineage in both formats: this W4A16 artifact in vLLM and the
verified IQ4_XS GGUF in llama.cpp. It is a separate common short-context
workload (--max-model-len 4096 for vLLM): cache-isolated, unique prompts,
greedy decoding, and 256 fixed visible output tokens per request. It is not a
replacement for the 640K capacity validation above.
At the normal 370 W board limit, W4A16 reaches 1,819 aggregate visible tok/s at B16—87% of its B32 result of 2,089 tok/s—making B16 the practical knee for this matched workload. B32 remains the absolute-throughput setting. The dashed 275 W lines are an optional efficiency overlay, not the default performance configuration: they preserve low-concurrency throughput but reduce W4A16 B16/B32 saturation throughput. The IQ4_XS power-limit difference is small by comparison.
These are hardware-, prompt-, sampling-, and runtime-specific measurements,
not universal model-performance claims. The exact matched methodology, raw
results, and both power-limit CSVs are in
benchmarks/rtx3090-matched-vllm-llamacpp.md.
The separately measured 640K production curve, including TPOT/TTFT and raw
five-repeat data, remains in
benchmarks/rtx3090-throughput-curve.md.
This validated 640K profile batches independent requests continuously. It does
not enable prefix caching by default: in vLLM 0.27.1, hybrid Mamba models
leave it opt-in because the required Mamba cache all mode is experimental.
Prefix caching only helps requests that share an identical token prefix; it
does not improve a batch of unrelated prompts. Add --enable-prefix-caching
only for a shared-prefix workload and soak-test the exact context/cache profile
you intend to use. A CUDA:0 smoke of this exact 640K profile entered the
experimental Mamba warm-up path and was not completed, so it is intentionally
not in the validated long-context command. A smaller-context lab profile may
use the flag, but it is a separate experimental deployment choice rather than
a property of the published 640K validation.
We evaluated 2,000 multiple-choice items (500 each) using vLLM likelihood scoring. This is quantization evidence for this artifact, not a replacement for NVIDIA's full source-model benchmark suite.
| Task | W4A16 |
|---|---|
| ARC-Challenge | 62.0% |
| HellaSwag | 67.8% |
| PIQA | 87.0% |
| Winogrande | 72.8% |
| Equal-weight aggregate | 72.4% |
These are standalone W4A16 measurements. They are not used for a cross-format
quality claim, because such a claim requires the same source checkpoint and
identical evaluation methodology. Full methodology:
benchmarks/quality-scorecard.md.
On the original 541-prompt Google IFEval set, evaluated by Google's reference strict/loose checker with a shared NVIDIA-formatted direct-answer prompt and greedy decoding:
| Format | Prompt strict | Instruction strict | Prompt loose | Instruction loose |
|---|---|---|---|---|
| This W4A16 artifact | 73.94% | 81.29% | 77.45% | 83.69% |
| bartowski GGUF IQ4_XS | 74.49% | 82.01% | 78.00% | 84.53% |
Both rows derive from the NVIDIA Nemotron 3.5 Lightning 30B-A3B BF16
checkpoint lineage, use the same 541 Google IFEval prompts, one shared
NVIDIA-formatted direct-answer prompt manifest, identical greedy generation
settings, and Google's reference evaluator. The measured difference on this
run is small (403/541 versus 400/541 prompt-strict); it should not be treated
as a general quantization ranking. See benchmarks/ifeval.md
for checkpoint, GGUF metadata, runtime hashes, prompt construction, and raw
result locations.
No directly comparable BF16 perplexity was measured. A source-BF16 comparison was not practical on the available 24 GB hardware, and no incompatible PPL result is presented as a degradation estimate. For this instruction/reasoning checkpoint, the paired 2,000-item scorecard is the directly measured quantization-quality evidence.
quantization.md: source revision, conversion settings, calibration methodology, and intentional compatibility changes.SHA256SUMS: SHA-256 checksums for the complete published payload.examples/start_vllm_3090_640k.sh: validated production launch profile.examples/client.py: OpenAI-compatible client example.Verify a download with:
bash
sha256sum -c SHA256SUMSvllm serve useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16 \ --quantization compressed-tensors \ --max-num-seqs 32curl http://localhost:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{ "model": "useful-quants/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-W4A16", "messages": [{"role": "user", "content": "Explain why the sky appears blue."}], "temperature": 1.0, "top_p": 0.95, "max_tokens": 512, "chat_template_kwargs": {"enable_thinking": true} }'bash examples/start_vllm_3090_640k.shsha256sum -c SHA256SUMSbenchmarks/quality-scorecard.md