heliophage

Nanbeige4.3-33B-W4A16-GPTQ

README

License: apache-2.0

What is quantized

Table with columns: Tensors, Precision, Notes
Tensors	Precision	Notes
`self_attn.{q,k,v,o}_proj`, all 52 layers	INT8 weights, BF16 activations (W8A16), group 128, symmetric	GPTQ
`mlp.{gate,up,down}_proj`, layers 0 and 51	W8A16, group 128, symmetric	GPTQ
`mlp.{gate,up,down}_proj`, layers 1–50	INT4 weights (W4A16), group 128, symmetric	GPTQ
`embed_tokens`, `lm_head` (166,912 × 7,168 each), norms	BF16	unchanged

vLLM fuses q/k/v into one matmul and gate/up into another, so every member of a fused group has the same precision.

Quick start (vLLM ≥ 0.27, 2× 24 GB)

bash

vllm serve heliophage/Nanbeige4.3-33B-W4A16-GPTQ \  --tensor-parallel-size 2 \  --max-model-len 65536 \  --gpu-memory-utilization 0.92 \  --enable-prefix-caching \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

Thinking is on by default. Turn it off per request with "chat_template_kwargs": {"enable_thinking": false}.
For multi-turn tool use and coding agents, Nanbeige recommends "chat_template_kwargs": {"preserve_thinking": true}.
Nanbeige's recommended sampling: agentic and tool use temperature=1.0, top_p=0.95; reasoning and chat temperature=0.6, top_p=0.95.
Keep the KV cache in BF16 on RTX 3090s. In vLLM 0.27.1 on Ampere (SM86), --kv-cache-dtype fp8 fails: FlashInfer crashes during startup (with fp8 and fp8_e5m2), and the Triton backend refuses it because native FP8 needs SM89+. On Ada/Hopper/Blackwell GPUs, FP8 KV roughly doubles capacity.

transformers can load this checkpoint (compressed-tensors decompresses it to BF16), but that needs about 66 GB of memory. Use vLLM to get the benefit of the quantized weights.

Serving performance (measured: vLLM 0.27.1, 2× RTX 3090 at a 350 W power limit, TP=2)

Started with the command above (BF16 KV cache). vLLM reports 8.78 GiB of KV cache per GPU, which is 88,480 tokens (1.35 concurrent requests at 64K context). Measured with vllm bench serve on random-token prompts, every request decoding exactly the stated number of tokens (scripts/bench_serve.sh; raw JSON in eval/serving/).

Table with columns: Workload, Concurrency, TTFT (median), Decode per user (median), Total output
Workload	Concurrency	TTFT (median)	Decode per user (median)	Total output
128 in / 256 out	1	64 ms	62.7 tok/s	61.9 tok/s
8K in / 256 out	1	3.5 s	58.6 tok/s	32.6 tok/s
32K in / 256 out	1	13.9 s	49.3 tok/s	13.4 tok/s
60K in / 256 out	1	40.4 s	39.9 tok/s	6.0 tok/s
1K in / 512 out	4	1.0 s	50.1 tok/s	186 tok/s
1K in / 512 out	16	2.5 s	34.4 tok/s	501 tok/s
1K in / 512 out	32	3.2 s	27.1 tok/s	655 tok/s

Prefill runs at about 2,100–2,350 tok/s up to 32K tokens of context. A functional check through the OpenAI API passed: reasoning output parsed into reasoning_content, an XML tool call parsed by qwen3_coder into tool_calls, and the tool result used in the follow-up answer (scripts/vllm_check.py).

Fidelity vs. the BF16 original

KL divergence of the next-token distribution against the BF16 model. This is what llama.cpp's --kl-divergence reports, and it's the metric used for AtomicChat's Nanbeige4.3-33B GGUFs. Lower is better; top-1 is how often the quant's most likely token matches BF16's. Every row is scored against the same BF16 reference, the same tokens and the same formula.

Agentic text: held-out Nemotron-Post-Training-v1 conversations (tool calling, code, math, chat) rendered with this model's chat template, so they include reasoning traces, XML tool calls and tool responses. 24 chunks × 4,096 tokens, second half of each chunk scored (49,128 tokens).

Table with columns: Model, Size, Mean KLD, Median KLD, p99 KLD, Same top-1, RMS Δp
Model	Size	Mean KLD	Median KLD	p99 KLD	Same top-1	RMS Δp
AtomicChat Q8_0 GGUF (reference point)	35.2 GB	0.0032	0.00005	0.016	98.59%	1.65%
AtomicChat AD-Q4_K_M GGUF	21.2 GB	0.0145	0.00038	0.148	96.95%	3.70%
This model	24.2 GB	0.0225	0.00046	0.263	96.52%	4.49%
Uniform W4A16 GPTQ (same calibration)	20.6 GB	0.0330	0.00076	0.376	95.83%	5.33%
AtomicChat Q4_K_M GGUF	20.1 GB	0.0322	0.00077	0.390	95.58%	5.35%

Neutral text: wikitext-2 test (English) and Chinese Wikipedia, plain prose with no chat markup, half each. Same chunking.

Table with columns: Model, Size, Mean KLD, Median KLD, p99 KLD, Same top-1, RMS Δp
Model	Size	Mean KLD	Median KLD	p99 KLD	Same top-1	RMS Δp
AtomicChat Q8_0 GGUF (reference point)	35.2 GB	0.0095	0.0015	0.081	96.42%	2.19%
AtomicChat AD-Q4_K_M GGUF	21.2 GB	0.0711	0.0167	0.905	90.14%	5.79%
This model	24.2 GB	0.1105	0.0332	1.335	87.01%	7.38%
AtomicChat Q4_K_M GGUF	20.1 GB	0.1108	0.0344	1.428	86.83%	7.23%
Uniform W4A16 GPTQ (same calibration)	20.6 GB	0.1598	0.0520	2.091	84.17%	8.78%

In short: of the vLLM-servable options here, this one stays closest to BF16. It beats plain Q4_K_M on agentic text and ties it on neutral text, and trails AtomicChat's importance-matrix AD-Q4_K_M, which needs llama.cpp.

How it was measured:

The BF16 reference was computed with transformers (Apple MPS, BF16). Its log-probabilities were stored in llama.cpp's --kl-divergence-base file format, so the GGUFs were scored by llama-perplexity --kl-divergence (llama.cpp 6050b46, Metal) against the identical reference. The compressed-tensors models were scored in transformers with a line-for-line port of llama.cpp's KLD statistics (scripts/kld_llamacpp.py).
Different backends add a small KLD floor to the GGUF rows: at most ~0.006 judging by Q8_0 (0.0095 here vs. 0.0032 on AtomicChat's own stand). That's small next to the gaps between the 4-bit rows.
Token streams and all logs are in eval/kld/; the summary is eval/kld/results.json.
KLD measures closeness to BF16, not task accuracy. No downstream benchmarks (lm-eval, AIME, SWE-bench) were run.

How it was made

Tool: llm-compressor 0.14.1.dev77+ge7a7016e (git main), compressed-tensors 0.19.1a20261023, transformers 5.20.2, PyTorch 2.14.1+cu130.
Algorithm: GPTQ, dampening_frac=0.1, actorder=static, symmetric INT weights, group size 128, sequential layer-by-layer calibration with error propagation (each layer is calibrated on the previous quantized layer's output). Hessians accumulated in FP32. Recipe: recipe.yaml.
Calibration (640 samples, 1.11 M tokens, ≤ 2,048 tokens each):
- 512 conversations from nvidia/Nemotron-Post-Training-Dataset-v1: 200 tool_calling (kept only if the whole conversation fits, so tool-call syntax is never truncated), 128 code, 64 math, 64 stem, 56 chat. Rendered with this model's own chat template (preserve_thinking=True, XML tool-call format), so calibration covers <think> traces, tool calls and tool responses (scripts/build_calib.py).
- 128 plain-text articles from Wikipedia, 64 English and 64 Chinese (scripts/build_calib_v2.py). Together with raising GPTQ dampening from 0.01 to 0.1, this cut neutral-text KLD by 12% versus an agentic-only, dampening-0.01 build, with agentic KLD unchanged within noise (the two changes weren't isolated).
Hardware: 2× RTX 3090 in data-parallel (torchrun, scripts/quantize_ddp.py, RECIPE=mixed DAMPENING=0.1). Weights were shared across ranks through /dev/shm and spilled to disk; the calibration samples were split between the GPUs, Hessians were reduced to the owning rank, and the GPTQ solves were split across GPUs. 89 minutes.
Mixed-precision choice: a uniform W4A16 build with the same calibration is 3.5 GB smaller but has ~45% more KLD on both corpora (table above). The extra 3.5 GB costs ~16% of KV-cache capacity on 2× 24 GB.

GPTQ in llm-compressor isn't bit-reproducible run to run. Repeat runs with identical settings differed in ~8–15% of INT4 values, with KLD within ±5% of each other.

Limitations

Same capabilities and limitations as the base model; see its card.
Fidelity was measured at a 4,096-token context. Long-context behaviour (the base model supports 512K) wasn't evaluated separately.
Needs two 24 GB GPUs (or one ≥ 32 GB GPU). It won't fit a single 24 GB card; use the single-GPU repo for that.

License and credits

The weights are derived from Nanbeige/Nanbeige4.3-33B by the Nanbeige team and are released under the same Apache-2.0 license (LICENSE). Calibration data came from nvidia/Nemotron-Post-Training-Dataset-v1 (CC BY 4.0) and Wikipedia (CC BY-SA 4.0); none of it is redistributed here. The GGUF comparison files are from AtomicChat/Nanbeige4.3-33B-GGUF.

bibtex

@misc{nanbeige43chat2026,  title        = {Nanbeige4.3-33B},  author       = {Nanbeige Team},  year         = {2026},  publisher    = {Hugging Face},  howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.3-33B}}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

heliophage

Model Tree

Base

Nanbeige/Nanbeige4.3-33B

Quantized

this model

Input Modalities

Text

Output Modalities