hsam007

Nanbeige4.3-33B-NVFP4

README

License: apache-2.0

Quantization details

Table with columns: Property, Configuration
Property	Configuration
Source checkpoint	`Nanbeige/Nanbeige4.3-33B`
Source revision	`c1093c501cc6460427c0ac634db7223d0450c0ee`
Architecture	`LlamaForCausalLM`, approximately 33.12B parameters
Method	Post-training quantization; no fine-tuning or gradient updates
Tool	NVIDIA Model Optimizer 0.47.0
Recipe	`NVFP4_DEFAULT_CFG`
Quantized operations	Attention and MLP linear projections, W4A4
Group size	16
Scales	FP8 E4M3 block scales and FP32 global scales
Higher-precision components	Token embeddings, `lm_head`, and normalization parameters
KV cache	Not quantized by this checkpoint
Export	Unified Hugging Face checkpoint, five Safetensors shards
Packed projection matrices	364

Calibration

Calibration used 1,024 examples and 412,992 tokens, with a maximum sequence length of 2,048 tokens. Batches of eight examples were grouped by length to reduce padding. Dataset revisions and sampling used a fixed seed of 42.

Table with columns: Calibration source, Examples, Coverage
Calibration source	Examples	Coverage
HuggingFaceH4/CodeAlpaca_20K	512	Coding instructions and responses
HuggingFaceH4/ultrachat_200k	256	General conversation
cais/mmlu, validation subsets	256	Science: 128; mathematics: 64; humanities, business, and medicine: 64

These datasets supplied representative inputs for measuring quantization scales. Their inclusion here does not represent a reported CodeAlpaca, UltraChat, or MMLU benchmark accuracy score. Calibration and the held-out diagnostic checks used separate examples.

Pinned dataset revisions:

CodeAlpaca: 798c567f69c8f4b12fc191015e59ee34e9afe00d
UltraChat: 8049631c405ae6576f93f445c6b8166f76f5505a
MMLU: c30699e8356da336a370243923dbaf21066bb9fe

Evaluation results

AIME 2026

Table with columns: Measurement, Observed result
Measurement	Observed result
Completed problems	30 / 30
Correct	28 / 30
Accuracy, one completed attempt per problem	93.33%
Request errors	0
Truncated generations	0

A Spot VM interruption stopped problem 30 during the first attempt. Its partial trace was preserved, and the interrupted attempt was restarted from the original question using the same seed and generation settings. The other 29 completed results were preserved unchanged. No answer hints were provided.

Protocol:

Dataset: MathArena/aime_2026, train split, revision d61f486805306e6e6b5913504c07763a99e1b112.
Zero-shot problem statements with an instruction to return the final integer inside \boxed{}.
One completed attempt per problem, temperature 0.6, top-p 0.95, seed 42 + problem_idx.
Maximum output budget: 262,144 tokens per problem.
enable_thinking=true and preserve_thinking=true.
Exact integer answer matching; no tools, code execution, external judge, or gold answers in the model input.

This is an independent NVFP4 evaluation. It does not reproduce the upstream publisher's full evaluation protocol, and there is no matched BF16 AIME run yet. The result should not be used to calculate an accuracy-retention percentage relative to the upstream model card.

GPQA Diamond

Table with columns: Benchmark, Accuracy
Benchmark	Accuracy
GPQA Diamond (partial run)	88.37%

This independent NVFP4 one-pass run was stopped before the full dataset completed. Responses without a parseable final choice, including output-limit truncations, are counted as incorrect; cancelled in-flight requests are unscored. This is a partial-run result, not a full-dataset score or a matched BF16 comparison.

Protocol: GPQA Diamond from the OpenAI simple-evals dataset, SHA-256 41d1213cd7a4998605a26c2798500652572007161b3a92817ba46b35befcd305; Inspect AI multiple-choice scoring with chain-of-thought prompting and exact final-choice matching. Answer choices were shuffled once with seed 42. Generation used temperature 0.6, top-p 0.95, seed 42, a 65,536-token output limit, and thinking enabled. One attempt per question; retries applied only to request errors. Gold answers were used only for scoring.

Held-out perplexity checks

We compared the original BF16 model with the packed NVFP4 checkpoint running in vLLM on 80 held-out examples, scoring 23,359 next-token predictions. Inputs were capped at 1,024 tokens per example. Lower perplexity is better.

Table with columns: Domain, Examples, BF16 perplexity, NVFP4 perplexity
Domain	Examples	BF16 perplexity	NVFP4 perplexity
Coding	32	20.613	19.870
General chat	16	8.522	8.448
Science	16	63.621	56.832
Mathematics	8	56.572	53.539
Humanities, business, and medicine	8	71.416	63.886
Aggregate, token-weighted	80	14.230	13.818

No perplexity regression was observed in this small diagnostic set. Lower perplexity here does not establish improved downstream accuracy or general BF16 parity. These are separate held-out checks, not standard full-suite benchmark scores.

The packed checkpoint also passed five short generation checks covering arithmetic, JSON output, a Python function, a science fact, and geography. Those checks verify basic loading and generation only.

Matched BF16 versus NVFP4 serving performance

Measured on one NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), using vLLM 0.32.0, native CUTLASS NVFP4, FlashAttention 2, and CUDA graphs for both formats. NVFP4 delivered about 2.0x the per-request decode rate in these workloads.

Single-request serving: median TTFT and median decode rate, where decode rate is 1000 / median TPOT (ms) and excludes prefill and the first token.

Table with columns: Input tokens, BF16 TTFT (ms), NVFP4 TTFT (ms), BF16 decode (tokens/s), NVFP4 decode (tokens/s)
Input tokens	BF16 TTFT (ms)	NVFP4 TTFT (ms)	BF16 decode (tokens/s)	NVFP4 decode (tokens/s)
128	55.4	22.6	21.9	44.1
1,024	183.2	70.5	21.9	44.1
4,096	739.5	262.5	21.7	43.1

Four concurrent requests: median TTFT and aggregate output throughput across all requests, rather than per-request decode rate.

Table with columns: Input tokens per request, BF16 TTFT (ms), NVFP4 TTFT (ms), BF16 aggregate (tokens/s), NVFP4 aggregate (tokens/s)
Input tokens per request	BF16 TTFT (ms)	NVFP4 TTFT (ms)	BF16 aggregate (tokens/s)	NVFP4 aggregate (tokens/s)
128	140.9	52.5	88.5	180.3
1,024	718.8	237.9	83.5	171.2
4,096	1891.3	792.6	68.4	141.2

Protocol: vllm bench serve, identical seeded synthetic prompts and tokenizer, 256 generated tokens per request with EOS ignored, 12 measured requests and two warmups per case, temperature 0, seed 42. Both servers used an 8,192-token context limit, an 8 GiB BF16 KV cache, four maximum sequences, an 8,192-token batch limit, and disabled prefix caching. Startup, compilation, and graph capture are excluded. TTFT is client-observed over server-local loopback. No competing GPU processes were detected at case boundaries. These are workload-specific serving measurements, not accuracy scores or maximum-throughput claims.

See performance-results.json for full medians, exploratory p95 values, and configuration.

A separate profiler audit confirmed the SM120 block-scaled FP4 GEMM path. A 1 GiB streaming-read hardware test reached approximately 1.53 TB/s; this hardware test does not measure whole-model DRAM throughput.

Additional evaluation

AIME 2026: 28/30, 93.33% in this independent one-pass run.
Matched BF16 versus NVFP4 serving comparison: complete, with CUDA graphs enabled for both formats.
GPQA Diamond (partial run): 88.37%; independent NVFP4 one-pass result, detailed above.
SWE-bench Verified: an NVFP4-only 20-task pilot is in progress; no final pilot score is reported yet.
SciCode: identified as a suitable next coding evaluation; no score is reported yet.

Run with vLLM

The tested runtime is vLLM 0.32.0. Use a compatible NVIDIA GPU and NVFP4-enabled runtime. Model memory, KV-cache memory, and available context length depend on hardware and serving settings.

bash

vllm serve hsam007/Nanbeige4.3-33B-NVFP4 \  --served-model-name Nanbeige4.3-33B-NVFP4 \  --host 127.0.0.1 \  --port 8000 \  --quantization modelopt_fp4 \  --dtype bfloat16 \  --max-model-len 32768 \  --gpu-memory-utilization 0.85 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --generation-config vllm \  --kernel-config '{"linear_backend":"cutlass","enable_flashinfer_autotune":false,"enable_jit_warmup":false,"enable_cutedsl_warmup":false}'

Example request:

bash

curl http://127.0.0.1:8000/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{    "model": "Nanbeige4.3-33B-NVFP4",    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],    "temperature": 0.6,    "top_p": 0.95,    "max_tokens": 4096,    "chat_template_kwargs": {      "enable_thinking": true,      "preserve_thinking": true    }  }'

The 4,096-token limit in this usage example is separate from the 262,144-token AIME evaluation budget. The source configuration advertises a 524,288-token context window; this release does not claim full-window quality validation or that every GPU can serve that length.

Attribution and license

The original model is developed by Nanbeige LLM Lab. This repository is a community quantized derivative and does not imply endorsement by the original authors.

The source model is released under Apache-2.0. See LICENSE and NOTICE. Please credit the original Nanbeige4.3-33B release when using this checkpoint.

Optimized vLLM kernels: ~1.4× faster single-stream decode

~1.4× faster single-stream decode with the optimized kernel package

47.8 → 66.3 tokens/s on one RTX PRO 6000 Blackwell Server Edition, compared with the untuned NVFP4 vLLM setup.

The weights in this repository do not automatically install the serving kernels. The optional nanbeige-vllm-kernels 0.1.1 wheel packages the validated worker, selected NVFP4 tactics and Python/Triton kernels. For a fresh Linux Python 3.12 environment with a compatible CUDA 13 compiler/toolchain already configured, install the optional pinned Python serving stack with [runtime]:

python3.12 -m venv .venv. .venv/bin/activatepython -m pip install 'nanbeige-vllm-kernels[runtime] @ https://huggingface.co/hsam007/Nanbeige4.3-33B-NVFP4/resolve/main/kernels/nanbeige_vllm_kernels-0.1.1-py3-none-any.whl'nanbeige-vllm doctor --jsonnanbeige-vllm serve hsam007/Nanbeige4.3-33B-NVFP4 --dry-runnanbeige-vllm serve hsam007/Nanbeige4.3-33B-NVFP4

If the tested runtime is already installed, install only the lightweight adapter:

python -m pip install 'https://huggingface.co/hsam007/Nanbeige4.3-33B-NVFP4/resolve/main/kernels/nanbeige_vllm_kernels-0.1.1-py3-none-any.whl'

The extra includes nvidia-cutlass-dsl[cu13]==4.8.0 for the selected CuTe DSL norm/quantization fusion. It does not install NVCC or a CUDA toolkit: set CUDA_HOME and make the compatible compiler available on PATH before first launch.

The validated profile is Linux/Python 3.12, vLLM 0.32.0 (V2 runner), Torch 2.13.0+cu130, FlashInfer 0.7.0.post1, Triton 3.7.1, NVIDIA CUTLASS DSL 4.8.0 with CUDA 13 support, and TP1 on an SM120 / 188-SM RTX PRO 6000 Blackwell Server Edition. The supplied preset uses 8192 context, at most 4 sequences, 8192 batched tokens, 8 GiB BF16 KV cache, BF16 output head and disabled prefix caching. First launch also needs writable compilation caches.

Normal vLLM options pass through. Changing context length removes the inherited fixed 8 GiB KV limit unless an explicit KV budget is supplied; longer contexts or larger batches are outside the measured preset and can use guarded original paths. A portable YAML/environment export for native vllm serve is available through nanbeige-vllm export-config --output ./nanbeige-config.

See installation, runtime requirements and scope. Install from the explicit wheel URL; no PyPI publication is implied. The existing model benchmark results describe their own evaluation settings and are unchanged by installing this optional adapter.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

hsam007

Model Tree

Base

Nanbeige/Nanbeige4.3-33B

Quantized

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quantization details

Table with columns: Property, Configuration
Property	Configuration
Source checkpoint	`Nanbeige/Nanbeige4.3-33B`
Source revision	`c1093c501cc6460427c0ac634db7223d0450c0ee`
Architecture	`LlamaForCausalLM`, approximately 33.12B parameters
Method	Post-training quantization; no fine-tuning or gradient updates
Tool	NVIDIA Model Optimizer 0.47.0
Recipe	`NVFP4_DEFAULT_CFG`
Quantized operations	Attention and MLP linear projections, W4A4
Group size	16
Scales	FP8 E4M3 block scales and FP32 global scales
Higher-precision components	Token embeddings, `lm_head`, and normalization parameters
KV cache	Not quantized by this checkpoint
Export	Unified Hugging Face checkpoint, five Safetensors shards
Packed projection matrices	364

Calibration

Table with columns: Calibration source, Examples, Coverage
Calibration source	Examples	Coverage
HuggingFaceH4/CodeAlpaca_20K	512	Coding instructions and responses
HuggingFaceH4/ultrachat_200k	256	General conversation
cais/mmlu, validation subsets	256	Science: 128; mathematics: 64; humanities, business, and medicine: 64

Pinned dataset revisions:

CodeAlpaca: 798c567f69c8f4b12fc191015e59ee34e9afe00d
UltraChat: 8049631c405ae6576f93f445c6b8166f76f5505a
MMLU: c30699e8356da336a370243923dbaf21066bb9fe

Evaluation results

AIME 2026

Table with columns: Measurement, Observed result
Measurement	Observed result
Completed problems	30 / 30
Correct	28 / 30
Accuracy, one completed attempt per problem	93.33%
Request errors	0
Truncated generations	0

Protocol:

Dataset: MathArena/aime_2026, train split, revision d61f486805306e6e6b5913504c07763a99e1b112.
Zero-shot problem statements with an instruction to return the final integer inside \boxed{}.
One completed attempt per problem, temperature 0.6, top-p 0.95, seed 42 + problem_idx.
Maximum output budget: 262,144 tokens per problem.
enable_thinking=true and preserve_thinking=true.
Exact integer answer matching; no tools, code execution, external judge, or gold answers in the model input.

GPQA Diamond

Table with columns: Benchmark, Accuracy
Benchmark	Accuracy
GPQA Diamond (partial run)	88.37%

Held-out perplexity checks

Table with columns: Domain, Examples, BF16 perplexity, NVFP4 perplexity
Domain	Examples	BF16 perplexity	NVFP4 perplexity
Coding	32	20.613	19.870
General chat	16	8.522	8.448
Science	16	63.621	56.832
Mathematics	8	56.572	53.539
Humanities, business, and medicine	8	71.416	63.886
Aggregate, token-weighted	80	14.230	13.818

Matched BF16 versus NVFP4 serving performance

Single-request serving: median TTFT and median decode rate, where decode rate is 1000 / median TPOT (ms) and excludes prefill and the first token.

Table with columns: Input tokens, BF16 TTFT (ms), NVFP4 TTFT (ms), BF16 decode (tokens/s), NVFP4 decode (tokens/s)
Input tokens	BF16 TTFT (ms)	NVFP4 TTFT (ms)	BF16 decode (tokens/s)	NVFP4 decode (tokens/s)
128	55.4	22.6	21.9	44.1
1,024	183.2	70.5	21.9	44.1
4,096	739.5	262.5	21.7	43.1

Four concurrent requests: median TTFT and aggregate output throughput across all requests, rather than per-request decode rate.

Table with columns: Input tokens per request, BF16 TTFT (ms), NVFP4 TTFT (ms), BF16 aggregate (tokens/s), NVFP4 aggregate (tokens/s)
Input tokens per request	BF16 TTFT (ms)	NVFP4 TTFT (ms)	BF16 aggregate (tokens/s)	NVFP4 aggregate (tokens/s)
128	140.9	52.5	88.5	180.3
1,024	718.8	237.9	83.5	171.2
4,096	1891.3	792.6	68.4	141.2

See performance-results.json for full medians, exploratory p95 values, and configuration.

Additional evaluation

AIME 2026: 28/30, 93.33% in this independent one-pass run.
Matched BF16 versus NVFP4 serving comparison: complete, with CUDA graphs enabled for both formats.
GPQA Diamond (partial run): 88.37%; independent NVFP4 one-pass result, detailed above.
SWE-bench Verified: an NVFP4-only 20-task pilot is in progress; no final pilot score is reported yet.
SciCode: identified as a suitable next coding evaluation; no score is reported yet.

Run with vLLM

The tested runtime is vLLM 0.32.0. Use a compatible NVIDIA GPU and NVFP4-enabled runtime. Model memory, KV-cache memory, and available context length depend on hardware and serving settings.

bash

vllm serve hsam007/Nanbeige4.3-33B-NVFP4 \  --served-model-name Nanbeige4.3-33B-NVFP4 \  --host 127.0.0.1 \  --port 8000 \  --quantization modelopt_fp4 \  --dtype bfloat16 \  --max-model-len 32768 \  --gpu-memory-utilization 0.85 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --generation-config vllm \  --kernel-config '{"linear_backend":"cutlass","enable_flashinfer_autotune":false,"enable_jit_warmup":false,"enable_cutedsl_warmup":false}'

vllm serve hsam007/Nanbeige4.3-33B-NVFP4 \  --served-model-name Nanbeige4.3-33B-NVFP4 \  --host 127.0.0.1 \  --port 8000 \  --quantization modelopt_fp4 \  --dtype bfloat16 \  --max-model-len 32768 \  --gpu-memory-utilization 0.85 \  --reasoning-parser qwen3 \  --enable-auto-tool-choice \  --tool-call-parser qwen3_coder \  --generation-config vllm \  --kernel-config '{"linear_backend":"cutlass","enable_flashinfer_autotune":false,"enable_jit_warmup":false,"enable_cutedsl_warmup":false}'

Example request:

bash

curl http://127.0.0.1:8000/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{    "model": "Nanbeige4.3-33B-NVFP4",    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],    "temperature": 0.6,    "top_p": 0.95,    "max_tokens": 4096,    "chat_template_kwargs": {      "enable_thinking": true,      "preserve_thinking": true    }  }'

curl http://127.0.0.1:8000/v1/chat/completions \  -H 'Content-Type: application/json' \  -d '{    "model": "Nanbeige4.3-33B-NVFP4",    "messages": [{"role": "user", "content": "Write a Python function that merges two sorted lists."}],    "temperature": 0.6,    "top_p": 0.95,    "max_tokens": 4096,    "chat_template_kwargs": {      "enable_thinking": true,      "preserve_thinking": true    }  }'

Attribution and license

The original model is developed by Nanbeige LLM Lab. This repository is a community quantized derivative and does not imply endorsement by the original authors.

The source model is released under Apache-2.0. See LICENSE and NOTICE. Please credit the original Nanbeige4.3-33B release when using this checkpoint.

Optimized vLLM kernels: ~1.4× faster single-stream decode

~1.4× faster single-stream decode with the optimized kernel package

47.8 → 66.3 tokens/s on one RTX PRO 6000 Blackwell Server Edition, compared with the untuned NVFP4 vLLM setup.

python3.12 -m venv .venv. .venv/bin/activatepython -m pip install 'nanbeige-vllm-kernels[runtime] @ https://huggingface.co/hsam007/Nanbeige4.3-33B-NVFP4/resolve/main/kernels/nanbeige_vllm_kernels-0.1.1-py3-none-any.whl'nanbeige-vllm doctor --jsonnanbeige-vllm serve hsam007/Nanbeige4.3-33B-NVFP4 --dry-runnanbeige-vllm serve hsam007/Nanbeige4.3-33B-NVFP4

python3.12 -m venv .venv. .venv/bin/activatepython -m pip install 'nanbeige-vllm-kernels[runtime] @ https://huggingface.co/hsam007/Nanbeige4.3-33B-NVFP4/resolve/main/kernels/nanbeige_vllm_kernels-0.1.1-py3-none-any.whl'nanbeige-vllm doctor --jsonnanbeige-vllm serve hsam007/Nanbeige4.3-33B-NVFP4 --dry-runnanbeige-vllm serve hsam007/Nanbeige4.3-33B-NVFP4

If the tested runtime is already installed, install only the lightweight adapter:

python -m pip install 'https://huggingface.co/hsam007/Nanbeige4.3-33B-NVFP4/resolve/main/kernels/nanbeige_vllm_kernels-0.1.1-py3-none-any.whl'

python -m pip install 'https://huggingface.co/hsam007/Nanbeige4.3-33B-NVFP4/resolve/main/kernels/nanbeige_vllm_kernels-0.1.1-py3-none-any.whl'