Recipe
- Body: NVFP4A16 (FP4 E2M1 weights, group size 16, 16-bit activations), GPTQ.
- 46 units kept in INT8 weight-only (symmetric, group size 128): 41 attention, 5 MLP.
lm_head INT8 weight-only (group size 128). Embeddings BF16.
- Format:
compressed-tensors (llm-compressor 0.15.0, compressed-tensors 0.20.1).
- No training. Calibration only (see "Calibration data and license").
Usage
The measurements ran vLLM's offline engine (vllm.LLM) with the engine arguments below. The server command with the
same arguments is:
vllm serve ThakiCloud/Qwen3-8B-NVFP4A16-INT8-Mixed-B200 --max-model-len 8192 --gpu-memory-utilization 0.90 \ --generation-config vllm
from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")out = client.chat.completions.create( model="ThakiCloud/Qwen3-8B-NVFP4A16-INT8-Mixed-B200", messages=[{"role": "user", "content": "๋ํ๋ฏผ๊ตญ์ ์๋๋ ์ด๋์ธ๊ฐ์?"}], temperature=0.0, max_tokens=256, extra_body={"chat_template_kwargs": {"enable_thinking": False}},)print(out.choices[0].message.content)
Every quality number on this card was produced with greedy decoding (temperature 0), the model's chat template with
enable_thinking=False, max_model_len 8192 and seed 0. The bundled generation_config.json keeps Qwen3's sampling
defaults (temperature 0.6, top-p 0.95, top-k 20). The measurements ignored that file (--generation-config vllm) and
set decoding per request.
Supported scope
Table with columns: Item, Verified, Not verified| Item | Verified | Not verified |
|---|
| GPU | NVIDIA B200 (one GPU) | any other GPU, including other Blackwell parts, Hopper and earlier |
| Serving engine | vLLM 0.29.0 (vllm/vllm-openai:v0.29.0, torch 2.13.0+cu130) | other vLLM versions, Transformers, TGI, SGLang, llama.cpp |
| Decoding | greedy, enable_thinking=False | thinking mode, sampling |
| Batch | latency measured at batch size 1 | throughput at larger batch sizes |
| Context | max_model_len 8192 | longer contexts |
The packed kernels (MarlinNvFp4 for the FP4 body, Marlin for the INT8 units and lm_head) were confirmed on B200 with
vLLM 0.29.0. On other stacks the same file may load through different kernels or fall back to float, with different
speed and accuracy.
Checkpoint identity
There are three different hash values, and each one covers something different.
Table with columns: Value, What is hashed, Value| Value | What is hashed | Value |
|---|
| Weight file SHA-256 | the bytes of model.safetensors (the same value Hugging Face shows as the LFS SHA-256) | 285c2530414baa623ad5f36cf081cb5cf148a81a172758eb3f41c68208181581 |
| Per-file SHA-256 list | the SHA256SUMS file in this repository: one SHA-256 line for each of the 7 files below | see file |
| Checkpoint ID | the SHA-256 of the SHA256SUMS file bytes. It is a hash of the hash list, not of any weight file | 59e12a5f0c94a3b824bb10e81dce01e9f97afebb7105b9ab871a574712cd23b5 |
The checkpoint ID covers 7 files: chat_template.jinja, config.json, generation_config.json, model.safetensors,
recipe.yaml, tokenizer.json and tokenizer_config.json. The cards (README.md, MODEL_CARD.ko.md) and
SHA256SUMS itself are not covered. To verify a download:
sha256sum -c SHA256SUMS # each of the 7 files matches the listsha256sum SHA256SUMS # 59e12a5fโฆ means the checkpoint ID matches
The checkpoint was built once, reloaded in a fresh pod and process for verification, and then evaluated. The
evaluation jobs fetched it by checkpoint ID and confirmed that no quantization code path ran during evaluation.
Results for this checkpoint (REL-81)
REL-81 is an evaluation set built after the paper, with zero item overlap with the paper's DEV-80, SEL-81 and
FINAL-81 sets: 4,694 items (Korean 1,530, English 914, numeric 1,319, reasoning 626, plus report-only Korean math 245 and
Korean in-domain 60), drawn from KMMLU, KoBEST, MMLU, HellaSwag, BIG-bench, BBH, HRM8K and a programmatic
numeric generator, after excluding every source item used in earlier experiments. Its gates were preregistered and sealed before this checkpoint was evaluated. It was run once,
with BF16 on the same items, decoding and GPU. Verdict: RELEASE_PASS, validity VALID.
Table with columns: Gate, Limit, This checkpoint, 95% interval, Status| Gate | Limit | This checkpoint | 95% interval | Status |
|---|
| Kernels (of 144 quantized linear layers) | MarlinNvFp4 98 / Marlin 46, no float fallback | 98 / 46, fallback 0 | โ | PASS |
| Weight-load memory (bytes / 2^30) | < 5.98 GiB | 5.917 GiB (6,353,372,672 bytes) | โ | PASS |
| TPOT vs BF16, same GPU, batch 1 | โค 1.1x | 0.927x (3.666 ms vs 3.956 ms) | โ | PASS |
| Korean retention vs BF16 | โฅ 0.90 | 0.926 (n = 839 BF16-correct items) | 0.908 to 0.944 | PASS |
| Tier retention vs BF16 | โฅ 0.90 | 0.930 (n = 3,108) | 0.921 to 0.939 | PASS |
| Overall accuracy change | โฅ โ3 pt | โ0.71 pt | โ1.60 to +0.19 pt | PASS |
| Numeric accuracy change | โฅ โ3 pt | โ2.20 pt | โ4.13 to โ0.27 pt | point-estimate PASS; non-inferiority NOT ESTABLISHED |
| Reasoning accuracy change | โฅ โ3 pt | 0.00 pt | โ2.87 to +2.87 pt | PASS |
| Format-error rise | โค +2 pt | 0.00 pt | โ0.43 to +0.43 pt | PASS |
Per-axis accuracy (BF16 โ this checkpoint): Korean 54.8% โ 55.0%, English 65.9% โ 65.3%, numeric 89.7% โ 87.5%,
reasoning 77.3% โ 77.3%. Report-only axes: Korean math โ1.63 pt (n = 245), Korean in-domain +3.3 pt (n = 60).
Batch-1 serving: TTFT p50 15.655 ms vs 15.45 ms, decode 272.8 tok/s vs 252.8 tok/s (BF16). A separate reload check on a
fresh pod measured a TPOT ratio of 0.926x.
Intervals are 95% (z = 1.96): Wald intervals for retention, paired per-item intervals for accuracy changes.
"Point-estimate PASS" means the gate decision, which was preregistered on point estimates. The interval columns are
reported so the strength of each pass is visible; they did not change any verdict.
Results reported in the paper (FINAL-81, a different build)
The paper's numbers come from a different build of the same recipe (checkpoint hash prefix 7e6fcc531386). That
build was not retained, and two builds of A81 in the paper already had different hashes, so this file is not
byte-identical to the one the paper measured. Do not combine the two tables.
Table with columns: Measure, Paper build on FINAL-81 (5,168 items)| Measure | Paper build on FINAL-81 (5,168 items) |
|---|
| Weight-load memory | 5.917 GiB (BF16 15.268 GiB, 61.2% lower) |
| Korean retention | 0.904 (n = 1,565 Korean items; gate 0.90, margin 0.0036) |
| Tier retention | 0.931 |
| Overall accuracy change | โ0.31 pt |
| TPOT vs BF16 | 0.939x |
| Batch-1 TTFT p50 / decode | 15.39 ms / 272.0 tok/s (BF16 15.25 ms / 255.4 tok/s) |
| Kernels | 98 MarlinNvFp4 / 46 Marlin, no float fallback |
Aggregates, the module-level allocation and a reproduction guide are in the results repository
ThakiCloud/B200-LowBit-Quantization-Benchmarks.
Limitations
- Numeric questions are the main known risk. REL-81 cannot rule out a numeric drop larger than 3 points.
- The Korean margin in the paper was narrow (0.36 points on 1,565 items). A rebuild can land on either side of the gate,
which is why this specific file was evaluated on its own.
- One base model, one GPU type, one serving engine version. Latency was measured at batch size 1 only.
- Quality was measured on multiple-choice and short-answer items aggregated from public benchmarks and a programmatic
numeric generator. It is not a safety, bias, long-form generation or domain evaluation.
- Thinking mode was never evaluated.
Calibration data and license
Calibration used 128 sequences of public English text: 43 from FineWeb-Edu, 43 from FineMath-4+ and 42 from an
OpenCoder algorithmic subset.
Table with columns: Source, Revision, Config / files, License| Source | Revision | Config / files | License |
|---|
HuggingFaceFW/fineweb-edu | 87f09149ef47 | sample-10BT, sample/10BT/000_00000.parquet | ODC-By v1.0, also subject to the Common Crawl Terms of Use |
HuggingFaceTB/finemath | e92b25a61673 | finemath-4plus, finemath-4plus/train-00000-of-00064.parquet | ODC-By v1.0, also subject to the Common Crawl Terms of Use |
OpenCoder-LLM/opc-annealing-corpus | 078e4bc1d397 | algorithmic_corpus, 1.jsonl and 10.jsonl | ODC-By |
No Korean-weighted calibration was used. No AI-Hub data was used for calibration or evaluation. The calibration text and
the evaluation items are not distributed in this repository.
This model is a quantized derivative of Qwen3-8B, released by the Qwen team under Apache-2.0. The weights in this
repository are released under Apache-2.0.
Citation
@misc{han2026beyondbitwidth, title = {Beyond Bitwidth: Deployment-Constrained Mixed-Precision Quantization for Multilingual LLM Inference on NVIDIA Blackwell}, author = {Han, Hyojung}, year = {2026}, note = {Preprint in preparation}}
Please also cite Qwen3.