What is quantized
Table with columns: Module, Format, Notes| Module | Format | Notes |
|---|
mlp.gate_proj, mlp.up_proj, mlp.down_proj (64 layers) | NVFP4 | E2M1, block 16, static block scales, FP8 E4M3 block scales |
lm_head | NVFP4 | same block format |
self_attn.q/k/v/o_proj (16 full-attention layers) | FP8 | E4M3 per-tensor, W8A8 |
linear_attn.in_proj_qkv, in_proj_z, out_proj (48 GDN layers) | FP8 | E4M3 per-tensor, W8A8 |
embed_tokens, vision tower (model.visual.*), conv1d, in_proj_a/b, A_log, dt_bias, norms | BF16 | 401 quantized modules total (193 NVFP4 + 208 FP8) |
Quantizer-level parity with the NVIDIA reference checkpoint was verified exhaustively:
1,699 quantizers compared, 0 differences (scripts/compare_recipe.py).
Requirements
Blackwell-class hardware (SM ≥ 100) for the NVFP4 half of the model. Supported
runtimes:
- vLLM ≥ 0.29 — detects the checkpoint as
modelopt_mixed automatically
- TensorRT-LLM
- Transformers is not supported for pre-quantized NVFP4/FP8 ModelOpt
checkpoints (5.x only implements on-the-fly quantization)
Usage
vLLM
vllm serve veladan/Arete-27B-NVFP4 \ --tensor-parallel-size 1 \ --max-model-len 32768
vLLM infers quantization=modelopt_mixed from hf_quant_config.json; passing
--quantization modelopt_fp4 also resolves to the mixed loader.
TensorRT-LLM
trtllm-serve veladan/Arete-27B-NVFP4 --max_batch_size 8
Quantization details
Table with columns: Property, Value| Property | Value |
|---|
| Method | Mixed precision: NVFP4 (W4A4, block 16, static block scales) + FP8 (W8A8, per-tensor) |
| Calibration algorithm | Local-Hessian with FP8 scale sweep for the NVFP4 weights; max for FP8 |
| Calibration data | 2,048 sequences × 2,048 tokens |
| Output format | Hugging Face safetensors + hf_quant_config.json + .quant_summary.txt |
Calibration corpus — deviation from the NVIDIA recipe
NVIDIA calibrated this recipe on the Nemotron post-training dataset
(nvidia/Nemotron-Post-Training-Dataset-v2, gated). That corpus was not accessible
during this build, so calibration used an open mixture instead:
Table with columns: Source, Share, Domain| Source | Share | Domain |
|---|
HuggingFaceTB/smoltalk | 50 % | general chat / instructions |
nvidia/OpenCodeInstruct | 20 % | code |
AI-MO/NuminaMath-CoT | 30 % | math / reasoning |
The quantization recipe (formats, block sizes, algorithms, excluded modules) is
identical to NVIDIA's; only the calibration corpus differs. This is recorded in
quantization_metadata.json.
How this checkpoint was produced
Everything below is reproducible with the scripts in the accompanying repository
(scripts/recipe.py, scripts/quantize.py, scripts/compare_recipe.py,
scripts/validate_quant.py).
Hardware: 1× NVIDIA RTX PRO 6000 Blackwell Max-Q (96 GB), 2× in total on the
build host. CUDA 13.2, driver 615.71.
Software: nvidia-modelopt==0.48.0, torch==2.15.0+cu132,
transformers==5.22.2, triton==3.8.0 (pinned — see note below),
flash-linear-attention==0.6.0.
Recipe (--preset nvidia-mixed): ModelOpt's
NVFP4_W4A4_WEIGHT_LOCAL_HESSIAN_CFG as the base — NVFP4 E2M1 weights with block
size 16 and static block scales, activations in FP4 with dynamic block scaling —
plus two overrides on top:
- self-attention (
q/k/v/o_proj) and linear-attention (GDN) projections
(in_proj_qkv, in_proj_z, out_proj) switched to FP8 E4M3 per-tensor
(num_bits=(4,3), axis=None), weight and input quantizers only.
lm_head enabled for NVFP4 (ModelOpt excludes it by default; NVIDIA
quantizes it).
Embeddings, the vision tower, conv1d, in_proj_a/b, A_log and dt_bias stay in
BF16, as in the reference checkpoint.
Calibration: the Local-Hessian algorithm (method=local_hessian,
fp8_scale_sweep=True) with layer-by-layer calibration enabled; it runs a max pass
over all quantizers first, then one Hessian-accumulation pass, then an MSE weight
search per block. 2,048 sequences of 2,048 tokens.
Command:
python scripts/quantize.py --preset nvidia-mixed \ --calib-kind nemotron --calib-samples 2048 --calib-seqlen 2048python scripts/compare_recipe.py --ref nvidia/Qwen3.8-27B-NVFP4 \ --summaries output/Arete-27B-NVFP4/.quant_summary.txt <ref>/.quant_summary.txtpython scripts/validate_quant.py --layers 5
Measured cost: model load 15 s, calibration tokenization 19 s, quantization
55 min, export 2 min.
Verification performed: quantizer-by-quantizer comparison against the NVIDIA
reference checkpoint (1,699 quantizers, 0 differences), tensor layout comparison for
14 module categories (identical dtype/shape for every weight and scale), and
weight reconstruction error against the BF16 source (table below).
Known deviations from the reference:
Table with columns: Item, NVIDIA reference, This checkpoint| Item | NVIDIA reference | This checkpoint |
|---|
| Calibration corpus | Nemotron post-training dataset (gated) | Open mix: smoltalk 50 %, OpenCodeInstruct 20 %, NuminaMath-CoT 30 % |
hf_quant_config.json exclude_modules | lists only mtp* | lists the 146 explicitly excluded modules (equivalent information; both encode the same quantization) |
| MTP layer | 1 layer (excluded from quantization) | none (Arete-27B has mtp_num_hidden_layers=0) |
Toolchain note: ModelOpt 0.48's NVFP4 dynamic-block fake-quantization kernel uses
Triton block pointers, which the Triton 3.9 compiler no longer accepts
(Block pointers have been removed in favor of the tensor descriptor API). The build
pins triton==3.8.0; with 3.9 the calibration aborts on the first forward pass.
Verification
Weight-level reconstruction error versus the BF16 source
(scripts/validate_quant.py, dequantizing sample tensors):
Table with columns: Format, Mean relative L2 error, Tensors sampled| Format | Mean relative L2 error | Tensors sampled |
|---|
| FP8 (attention projections) | 0.0266 | 5 |
NVFP4 (MLP, lm_head) | 0.0874 | 3 |
For reference, a plain NVFP4-with-max-calibration attempt of the same model gave
0.0950 on the MLP tensors: the Local-Hessian calibration used here is measurably
better, which is why the NVIDIA recipe was adopted.
Smoke evaluation (arc_easy, 200 questions, 0-shot, served by vLLM): 0.825.
Full benchmark numbers are pending — see Evaluation.
Evaluation
Both checkpoints were served side by side with vLLM 0.29 on identical hardware and
evaluated with lm-evaluation-harness
using the same settings (MMLU: 5-shot via /v1/completions; GSM8K 5-shot and IFEval
0-shot via /v1/chat/completions with the model's chat template, greedy decoding).
Table with columns: Benchmark, Arete-27B (BF16), Arete-27B-NVFP4, Δ| Benchmark | Arete-27B (BF16) | Arete-27B-NVFP4 | Δ |
|---|
| MMLU (14,042 q, 5-shot, acc) | 0.8336 | 0.8220 | −0.0115 |
| GSM8K (1,319 q, 5-shot, strict) | 0.4041 | 0.4185 | +0.0144 |
| GSM8K (flexible extract) | 0.6073 | 0.6262 | +0.0190 |
| IFEval (541 prompts, prompt-level strict) | 0.3161 | 0.3142 | −0.0018 |
| IFEval (instruction-level loose) | 0.5132 | 0.5024 | −0.0108 |
Averaged over the 64 MMLU subjects the quantized model is −0.0097 below BF16,
with per-subject standard deviation comparable to the difference. GSM8K moves in the
opposite direction, i.e. within run-to-run noise for a benchmark of this size
(σ ≈ 0.014).
Reading of these numbers: the only consistent, measurable degradation is a
~1 point MMLU drop; conversation-style and math accuracy are unchanged within noise.
This is the expected profile of the NVIDIA recipe, which keeps attention in FP8 and
only pushes the MLP weights to 4 bits.
Not yet measured: GPQA Diamond, Terminal-Bench, AA-LCR, MMMU-Pro, SciCode and
IFBench — the benchmarks NVIDIA lists for the reference checkpoint. They require
additional harnesses (agentic execution, multimodal inputs), so the comparison
above is text-only.
Limitations
- Quantization is lossy. Weight error is ~9 % (NVFP4) and ~3 % (FP8) relative L2;
end-to-end effect on reasoning-heavy tasks is not yet measured.
- Calibrated on English chat/code/math data. Other languages or specialized domains
may degrade relative to the BF16 model.
- The vision tower remains in BF16 and was not calibrated with image inputs; image
understanding inherits the base model without quantization-specific tuning.
- Outputs may contain biases, inaccuracies or unsafe content inherited from the base
model. Not intended for high-stakes use without human oversight.
Ethical considerations & safety
This is a compressed derivative of NousResearch/Arete-27B and inherits its
behaviour, including biases and failure modes. No safety tuning was performed or
removed during quantization. Users are responsible for evaluating fitness for
purpose, complying with applicable law, and disclosing that outputs are
machine-generated.
Citation
@misc{arete27b_nvfp4, title = {Arete-27B-NVFP4}, author = {veladan}, year = {2025}, note = {Mixed NVFP4/FP8 quantization of NousResearch/Arete-27B via NVIDIA Model Optimizer}, url = {https://huggingface.co/veladan/Arete-27B-NVFP4}}
License
Released under the Apache License 2.0 (see LICENSE), consistent with the
upstream model. The base model's terms
(https://huggingface.co/NousResearch/Arete-27B) continue to apply to any derivative
work. Calibration datasets are used only for calibration and are not redistributed.