evanrodgers

Nanbeige4.3-33B-W4A16-G128

README

License: apache-2.0

Model details

Table with columns: Item, Value
Item	Value
Base model	`Nanbeige/Nanbeige4.3-33B` (instruction-tuned chat / thinking)
Architecture	`LlamaForCausalLM`, 52 layers, hidden 7168, 56 attention heads / 8 KV heads, head_dim 128, FFN 22016, vocab 166,912, 524,288 max positions, RoPE θ = 1.2e8
Parameters	33.12B (all retained; quantization changes precision, not parameter count)
Quantized modules	All `Linear` layers except `lm_head`
Scheme	GPTQ W4A16, INT4 symmetric, group size 128, static activation ordering
Format	compressed-tensors `pack-quantized` (auto-detected by vLLM; no `--quantization` flag needed)
Calibration	512 windows × 2048 tokens = 1.05M tokens, rendered through the model's own chat template; code 40% (Magicoder-OSS-Instruct-75K), tool 20% (glaive-function-calling-v2), chat 20% (ultrachat_200k), Chinese 20% (evol-instruct-chinese), shuffled
GPTQ settings	`block_size=128`, `dampening_frac=0.01`, `actorder="static"`, sequential per-decoder-layer calibration with CPU offload
Output	2 shards, 20.63 GB of weight data
Quantization cost	2.94 h on one 64 GB CMP 170HX
Toolchain	llmcompressor 0.14.0.1, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.13.0

Quickstart (vLLM)

Recent vLLM versions auto-detect the compressed-tensors config. This build was validated with vLLM 0.31.0:

bash

pip install "vllm==0.31.0"

bash

vllm serve evanrodgers/Nanbeige4.3-33B-W4A16-G128 \  --served-model-name nanbeige4.3-33b \  --max-model-len 32768 \  --gpu-memory-utilization 0.90 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder

Notes:

Add --tensor-parallel-size N to shard across GPUs. With 8 KV heads, values up to 8 divide cleanly.
--max-model-len 32768 is a safe starting point for a 40 GB card. The model supports 524,288 positions, but KV cache is ~208 KiB per token in BF16 (~104 GiB for a full 512K request), so full context needs multiple large GPUs. We validated 32K on a single 40 GB card and a 262K configuration on 2×64 GB GPUs (PP=2); we did not run full-length quality evaluations.
FP8 KV cache did not work in our validation: --kv-cache-dtype fp8 was unreliable, so leave the KV cache at the default BF16 and do not add that flag.
--reasoning-parser qwen3 splits the <think>…</think> trace into reasoning_content and leaves the answer in content.
--tool-call-parser qwen3_coder matches the base model's recommended tool-calling setup (tool_call_format="xml").

Python client

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create(    model="nanbeige4.3-33b",    messages=[{"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}],    temperature=0.6,    top_p=0.95,    max_tokens=32768,    extra_body={        "chat_template_kwargs": {            "enable_thinking": True,      # set False for non-thinking mode            "preserve_thinking": False,   # True for multi-turn agent/tool workflows        }    },) print(resp.choices[0].message.reasoning_content)  # thinking trace, if enabledprint(resp.choices[0].message.content)            # final answer

Recommended sampling settings from the base model:

Table with columns: Scenario, Temperature, Top-p, Max new tokens
Scenario	Temperature	Top-p	Max new tokens
Agentic and tool use	1.0	0.95	65,536
Reasoning and chat	0.6	0.95	262,144

For tool use, pass tools in the request and keep chat_template_kwargs.tool_call_format at its default "xml" for best tool-calling performance ("json" is also supported for compatibility).

Quantization recipe

The exact llmcompressor recipe is included as recipe.yaml:

yaml

default_stage:  default_modifiers:    GPTQModifier:      targets: [Linear]      ignore: [lm_head]      scheme: W4A16      block_size: 128      dampening_frac: 0.01      actorder: static

The calibration set is intentionally deployment-shaped: code, tool-call trajectories, general chat, and Chinese instructions, all rendered with this model's chat template rather than raw text. Calibration used 512 windows of 2048 tokens, shuffled, with one decoder layer resident on the GPU at a time (CPU offload between layers) to fit a single 64 GB card.

Minimal reproduction sketch:

python

from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import GPTQModifierfrom llmcompressor.recipe import Recipe recipe = Recipe.from_modifiers(    GPTQModifier(        targets="Linear",        scheme="W4A16",        ignore=["lm_head"],        block_size=128,        dampening_frac=0.01,        actorder="static",    )) oneshot(    model=model,                      # Nanbeige4.3-33B in BF16    tokenizer=tokenizer,    recipe=recipe,    dataset="json",    dataset_path="calib/n512_instruct",    splits="train",    text_column="text",    num_calibration_samples=512,    batch_size=1,    max_seq_length=2048,    shuffle_calibration_samples=True,    sequential_targets=["LlamaDecoderLayer"],    sequential_offload_device="cpu",    tie_word_embeddings=False,    save_compressed=True,    output_dir="Nanbeige4.3-33B-W4A16-G128",)

Limitations

Weight-only 4-bit quantization can regress relative to BF16, especially on long-context reasoning, agentic tool use, and code repair. Test your workload.
Long context is memory-bound. The weights are ~20.6 GB, but a 512K-token KV cache needs ~104 GiB in BF16. Use --max-model-len appropriate for your hardware.
No full-suite validation. We did not run the upstream agentic benchmark suite or full-context evaluations on this build; validate on your own workload.
Transformers generate(): the upstream config.json ships with use_cache: false. This does not affect vLLM, but if you use Transformers directly, pass use_cache=True to generate() (or set it in the config) to avoid recomputing the prefix on every step.
Intended for vLLM / compressed-tensors. Loading the packed INT4 weights through plain Transformers requires a recent transformers + compressed-tensors build.

License

Apache-2.0, inherited from the base model. See LICENSE. The base model's terms continue to apply.

Citation

If you use this quantized build, please cite the base model and GPTQ:

bibtex

@misc{nanbeige43chat2026,  title        = {Nanbeige4.3-33B},  author       = {Nanbeige Team},  year         = {2026},  publisher    = {Hugging Face},  howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.3-33B}}} @article{frantar2022gptq,  title   = {GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers},  author  = {Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},  journal = {arXiv:2210.17323},  year    = {2022}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

evanrodgers

Model Tree

Base

Nanbeige/Nanbeige4.3-33B

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model details

Table with columns: Item, Value
Item	Value
Base model	`Nanbeige/Nanbeige4.3-33B` (instruction-tuned chat / thinking)
Architecture	`LlamaForCausalLM`, 52 layers, hidden 7168, 56 attention heads / 8 KV heads, head_dim 128, FFN 22016, vocab 166,912, 524,288 max positions, RoPE θ = 1.2e8
Parameters	33.12B (all retained; quantization changes precision, not parameter count)
Quantized modules	All `Linear` layers except `lm_head`
Scheme	GPTQ W4A16, INT4 symmetric, group size 128, static activation ordering
Format	compressed-tensors `pack-quantized` (auto-detected by vLLM; no `--quantization` flag needed)
Calibration	512 windows × 2048 tokens = 1.05M tokens, rendered through the model's own chat template; code 40% (Magicoder-OSS-Instruct-75K), tool 20% (glaive-function-calling-v2), chat 20% (ultrachat_200k), Chinese 20% (evol-instruct-chinese), shuffled
GPTQ settings	`block_size=128`, `dampening_frac=0.01`, `actorder="static"`, sequential per-decoder-layer calibration with CPU offload
Output	2 shards, 20.63 GB of weight data
Quantization cost	2.94 h on one 64 GB CMP 170HX
Toolchain	llmcompressor 0.14.0.1, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.13.0

Quickstart (vLLM)

Recent vLLM versions auto-detect the compressed-tensors config. This build was validated with vLLM 0.31.0:

bash

pip install "vllm==0.31.0"

bash

vllm serve evanrodgers/Nanbeige4.3-33B-W4A16-G128 \  --served-model-name nanbeige4.3-33b \  --max-model-len 32768 \  --gpu-memory-utilization 0.90 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder

vllm serve evanrodgers/Nanbeige4.3-33B-W4A16-G128 \  --served-model-name nanbeige4.3-33b \  --max-model-len 32768 \  --gpu-memory-utilization 0.90 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder

Notes:

Add --tensor-parallel-size N to shard across GPUs. With 8 KV heads, values up to 8 divide cleanly.
--max-model-len 32768 is a safe starting point for a 40 GB card. The model supports 524,288 positions, but KV cache is ~208 KiB per token in BF16 (~104 GiB for a full 512K request), so full context needs multiple large GPUs. We validated 32K on a single 40 GB card and a 262K configuration on 2×64 GB GPUs (PP=2); we did not run full-length quality evaluations.
FP8 KV cache did not work in our validation: --kv-cache-dtype fp8 was unreliable, so leave the KV cache at the default BF16 and do not add that flag.
--reasoning-parser qwen3 splits the <think>…</think> trace into reasoning_content and leaves the answer in content.
--tool-call-parser qwen3_coder matches the base model's recommended tool-calling setup (tool_call_format="xml").

Python client

python

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create(    model="nanbeige4.3-33b",    messages=[{"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}],    temperature=0.6,    top_p=0.95,    max_tokens=32768,    extra_body={        "chat_template_kwargs": {            "enable_thinking": True,      # set False for non-thinking mode            "preserve_thinking": False,   # True for multi-turn agent/tool workflows        }    },) print(resp.choices[0].message.reasoning_content)  # thinking trace, if enabledprint(resp.choices[0].message.content)            # final answer

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY") resp = client.chat.completions.create(    model="nanbeige4.3-33b",    messages=[{"role": "user", "content": "Which number is bigger, 9.11 or 9.8?"}],    temperature=0.6,    top_p=0.95,    max_tokens=32768,    extra_body={        "chat_template_kwargs": {            "enable_thinking": True,      # set False for non-thinking mode            "preserve_thinking": False,   # True for multi-turn agent/tool workflows        }    },) print(resp.choices[0].message.reasoning_content)  # thinking trace, if enabledprint(resp.choices[0].message.content)            # final answer

Recommended sampling settings from the base model:

Table with columns: Scenario, Temperature, Top-p, Max new tokens
Scenario	Temperature	Top-p	Max new tokens
Agentic and tool use	1.0	0.95	65,536
Reasoning and chat	0.6	0.95	262,144

For tool use, pass tools in the request and keep chat_template_kwargs.tool_call_format at its default "xml" for best tool-calling performance ("json" is also supported for compatibility).

Quantization recipe

The exact llmcompressor recipe is included as recipe.yaml:

yaml

default_stage:  default_modifiers:    GPTQModifier:      targets: [Linear]      ignore: [lm_head]      scheme: W4A16      block_size: 128      dampening_frac: 0.01      actorder: static

default_stage:  default_modifiers:    GPTQModifier:      targets: [Linear]      ignore: [lm_head]      scheme: W4A16      block_size: 128      dampening_frac: 0.01      actorder: static

Minimal reproduction sketch:

python

from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import GPTQModifierfrom llmcompressor.recipe import Recipe recipe = Recipe.from_modifiers(    GPTQModifier(        targets="Linear",        scheme="W4A16",        ignore=["lm_head"],        block_size=128,        dampening_frac=0.01,        actorder="static",    )) oneshot(    model=model,                      # Nanbeige4.3-33B in BF16    tokenizer=tokenizer,    recipe=recipe,    dataset="json",    dataset_path="calib/n512_instruct",    splits="train",    text_column="text",    num_calibration_samples=512,    batch_size=1,    max_seq_length=2048,    shuffle_calibration_samples=True,    sequential_targets=["LlamaDecoderLayer"],    sequential_offload_device="cpu",    tie_word_embeddings=False,    save_compressed=True,    output_dir="Nanbeige4.3-33B-W4A16-G128",)

from llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import GPTQModifierfrom llmcompressor.recipe import Recipe recipe = Recipe.from_modifiers(    GPTQModifier(        targets="Linear",        scheme="W4A16",        ignore=["lm_head"],        block_size=128,        dampening_frac=0.01,        actorder="static",    )) oneshot(    model=model,                      # Nanbeige4.3-33B in BF16    tokenizer=tokenizer,    recipe=recipe,    dataset="json",    dataset_path="calib/n512_instruct",    splits="train",    text_column="text",    num_calibration_samples=512,    batch_size=1,    max_seq_length=2048,    shuffle_calibration_samples=True,    sequential_targets=["LlamaDecoderLayer"],    sequential_offload_device="cpu",    tie_word_embeddings=False,    save_compressed=True,    output_dir="Nanbeige4.3-33B-W4A16-G128",)

Limitations

Weight-only 4-bit quantization can regress relative to BF16, especially on long-context reasoning, agentic tool use, and code repair. Test your workload.
Long context is memory-bound. The weights are ~20.6 GB, but a 512K-token KV cache needs ~104 GiB in BF16. Use --max-model-len appropriate for your hardware.
No full-suite validation. We did not run the upstream agentic benchmark suite or full-context evaluations on this build; validate on your own workload.
Transformers generate(): the upstream config.json ships with use_cache: false. This does not affect vLLM, but if you use Transformers directly, pass use_cache=True to generate() (or set it in the config) to avoid recomputing the prefix on every step.
Intended for vLLM / compressed-tensors. Loading the packed INT4 weights through plain Transformers requires a recent transformers + compressed-tensors build.

License

Apache-2.0, inherited from the base model. See LICENSE. The base model's terms continue to apply.

Citation

If you use this quantized build, please cite the base model and GPTQ:

bibtex

@misc{nanbeige43chat2026,  title        = {Nanbeige4.3-33B},  author       = {Nanbeige Team},  year         = {2026},  publisher    = {Hugging Face},  howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.3-33B}}} @article{frantar2022gptq,  title   = {GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers},  author  = {Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},  journal = {arXiv:2210.17323},  year    = {2022}}

@misc{nanbeige43chat2026,  title        = {Nanbeige4.3-33B},  author       = {Nanbeige Team},  year         = {2026},  publisher    = {Hugging Face},  howpublished = {\url{https://huggingface.co/Nanbeige/Nanbeige4.3-33B}}} @article{frantar2022gptq,  title   = {GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers},  author  = {Frantar, Elias and Ashkboos, Saleh and Hoefler, Torsten and Alistarh, Dan},  journal = {arXiv:2210.17323},  year    = {2022}}