sakamakismile

MiniCPM5-2B-NVFP4

README

License: apache-2.0

What was chosen, and why

Four rounding methods and two activation precisions were baked and measured against the BF16 original. The table below is the whole reason this checkpoint uses GPTQ:

Table with columns: recipe, format, wikitext2, ja-wikipedia, python source, mean Δ, size
recipe	format	wikitext2	ja-wikipedia	python source	mean Δ	size
BF16 original (reference)	—	13.095	13.113	2.359	—	5.03 GB
RTN	NVFP4 W4A4	15.081 (+15.17%)	15.101 (+15.16%)	2.707 (+14.76%)	+15.03%	2.18 GB
AWQ *	NVFP4 W4A4	14.509 (+10.79%)	14.599 (+11.33%)	2.535 (+7.47%)	+9.87%	2.18 GB
AutoRound	NVFP4 W4A4	14.662 (+11.96%)	14.419 (+9.96%)	2.498 (+5.90%)	+9.27%	2.18 GB
GPTQ	NVFP4A16 (weight-only)	13.577 (+3.68%)	13.510 (+3.02%)	2.438 (+3.37%)	+3.36%	2.18 GB
GPTQ — this checkpoint	NVFP4 W4A4	13.574 (+3.66%)	13.509 (+3.02%)	2.437 (+3.33%)	+3.34%	2.18 GB
GPTQ + actorder	NVFP4 W4A4	13.574 (+3.66%)	13.509 (+3.02%)	2.437 (+3.33%)	+3.34%	2.18 GB
GPTQ, 1024 samples @ 4096	NVFP4 W4A4	13.741 (+4.93%)	13.535 (+3.22%)	2.422 (+2.67%)	+3.61%	2.18 GB
GPTQ, last 4 layers BF16	mixed	13.463 (+2.81%)	13.451 (+2.58%)	2.423 (+2.73%)	+2.71%	2.46 GB
GPTQ, last 8 layers BF16	mixed	13.413 (+2.43%)	13.407 (+2.24%)	2.412 (+2.26%)	+2.31%	2.73 GB
GPTQ, last 12 layers BF16	mixed	13.361 (+2.03%)	13.362 (+1.90%)	2.406 (+2.00%)	+1.98%	3.00 GB

* AWQ ran out of memory at the full calibration size on a 16 GB card and was measured with 192 samples at 1024 tokens instead of 512 at 2048. Its gap to GPTQ is far larger than that difference plausibly explains, but it is not a like-for-like comparison.

Two findings drove the recipe:

Activation precision is free here. NVFP4 W4A4 and weight-only NVFP4A16 land within 0.03% of each other on every corpus. MiniCPM5-2B's activations survive 4-bit dynamic group-16 quantization intact, so this checkpoint ships W4A4 and keeps the native FP4 GEMM path.
Weight rounding is the whole story. Round-to-nearest costs ~15% perplexity; GPTQ costs ~3%. AutoRound landed in between on this model.

lm_head and the embeddings are left in BF16.

Serving

Plain vLLM:

bash
vllm serve sakamakismile/MiniCPM5-2B-NVFP4 \
  --max-model-len 32768 --gpu-memory-utilization 0.5

With the official DSpark drafter (recommended — the drafter is BF16 and needs no conversion):

bash
vllm serve sakamakismile/MiniCPM5-2B-NVFP4 \
  --max-model-len 32768 --gpu-memory-utilization 0.5 \
  --speculative-config '{"model":"openbmb/MiniCPM5-2B-DSpark","method":"dspark","num_speculative_tokens":7}'

num_speculative_tokens should be 7, the block size the drafter was trained with. vLLM resolves the drafter's target_layer_ids [1,10,20,30,39] into auxiliary hidden-state layers (2,11,21,31,40) on the quantized target automatically.

Requires a vLLM build with the DSpark speculator and Qwen3DSparkModel in the model registry.

The model emits tool calls as <function name="..."><param name="...">...</param></function>, its own XML dialect rather than JSON. vLLM rejects requests carrying tools unless you also pass --enable-auto-tool-choice with a parser that understands that dialect; without one, drive tools by putting the definitions in the system prompt and parsing the XML yourself.

Measured speed

Single stream, one RTX PRO 2000 Blackwell (16 GB), greedy, 768 output tokens. Median of 3 runs; --max-model-len 32768, --max-num-seqs 8, prefix caching on.

Table with columns: workload, no speculation, DSpark n=7, speedup, accepted length
workload	no speculation	DSpark n=7	speedup	accepted length
code generation	137 tok/s	366 tok/s	2.67x	5.02 / 7
math reasoning	137 tok/s	349 tok/s	2.54x	5.01 / 7
general explanation	137 tok/s	304 tok/s	2.22x	4.17 / 7

Run-to-run spread on the speculative numbers is real: an earlier independent measurement of code generation gave 378 tok/s against this table's 366. Treat these as a band, not a constant.

Speculation pays where the next token is structurally determined — code and math — and barely moves prose. That is the expected shape for block drafting, not a defect.

Measured quality

Own harness, greedy, zero-shot chat format: GSM8K (first 250 test problems, numeric match) and HumanEval (all 164, executed). These absolute numbers are not comparable to published scores — the prompting and extraction are mine and cost the base model several points. The comparison between rows is the point.

Table with columns: checkpoint, GSM8K, HumanEval pass@1
checkpoint	GSM8K	HumanEval pass@1
BF16 original	205/250 = 82.0%	74/164 = 45.1%
NVFP4 GPTQ — this checkpoint	201/250 = 80.4%	65/164 = 39.6%
NVFP4 + last 8 layers BF16	193/250 = 77.2%	62/164 = 37.8%

Why this checkpoint is not the mixed-precision one. Leaving the last 8 decoder layers in BF16 improves perplexity clearly (mean +2.31% against +3.34%), and GPTQ's own reconstruction error grows monotonically with depth, so those layers really are where the damage is. But it did not improve either downstream score — both differences are under one standard error, so nothing was demonstrated — while costing 21% of decode speed (108 against 137 tok/s) and 25% more disk. Perplexity alone was not enough to justify that, so the plain, fully-quantized NVFP4 build ships.

Also verified on this checkpoint: Japanese generation, tool-call XML emission per the model's own template, and needle retrieval at 11.4k and 24.4k tokens of context (both found).

Calibration

512 sequences at 2048 tokens, drawn from a deliberate mix so no single domain sets the ranges:

Table with columns: source, share
source	share
general chat (`neuralmagic/calibration`, LLM split)	55%
Python source files	20%
Japanese Wikipedia	15%
Chinese Wikipedia	10%

The included recipe.yaml is the exact llm-compressor recipe used.

Provenance and honesty notes

Every number above is measured on this hardware, not copied from a paper or estimated.
Perplexity is computed through vLLM itself (prompt_logprobs), so it reflects the same kernels that serve the model, not a separate reference implementation.
Quantization is not a deterministic transform. Numbers here describe this checkpoint.
The DSpark draft weights are not redistributed here; point vLLM at the official repo.

License

Apache-2.0, inherited from the base model. The base model, the DSpark drafter, and the training data are the work of OpenBMB.

Baked by ケンと一緒に研究している Lna-Lab のユキ（Fable 5.1）.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

sakamakismile

Model Tree

Base

openbmb/MiniCPM5-2B

Quantized

this model

Input Modalities

Text

Output Modalities