NatdhanaiPython

ThaiLLM-30B-NVFP4

README

License: apache-2.0

Serving

bash
# vLLM (Blackwell GPU, e.g. DGX Spark with nvcr.io/nvidia/vllm:26.05.post1-py3)
vllm serve NatdhanaiPython/ThaiLLM-30B-NVFP4 \
  --quantization modelopt \
  --gpu-memory-utilization 0.70 \
  --attention-backend flashinfer
# vLLM auto-selects native FLASHINFER_CUTLASS NVFP4 MoE kernels on SM121/SM100.
# For extra KV headroom in production add: --kv-cache-dtype fp8

Benchmarks (paired BF16 vs NVFP4, identical questions, seed 0)

Table with columns: Benchmark (n), BF16, NVFP4, Δ (pt), Paired p, Verdict
Benchmark (n)	BF16	NVFP4	Δ (pt)	Paired p	Verdict
ThaiExam (letter MC, 565)	0.6195	0.6142	−0.53	0.79	not significant
Belebele-TH (900)	0.7700	0.7656	−0.44	—	noise
XCOPA-TH (500)	0.6400	0.6340	−0.60	—	noise
XNLI-TH (2,490)	0.4707	0.4578	−1.29	—	≤1.3σ
Thai MC pooled (3,890)	—	—	−1.00	0.13	not significant
MMLU 5-shot @50/subj (2,850)	0.8168	0.8028	−1.40	0.0017	significant
HellaSwag (10,042)	0.6003	0.5941	−0.62	<0.001	significant, small
ARC-Challenge (1,172)	0.5614	0.5597	−0.17	—	noise
WinoGrande (1,267)	0.7395	0.7285	−1.10	—	≤0.9σ
All MC pooled (19,786)	—	—	−0.81	<1e-4	significant, small

Table with columns: Fidelity / LM quality, BF16, NVFP4
Fidelity / LM quality	BF16	NVFP4
Thai Wikipedia bits/byte ↓	0.2680	0.2822
WikiText-2 bits/byte ↓	0.5660	0.5819
Thai top-1 next-token agreement	—	92.0 % (32,424 positions)
English top-1 agreement	—	88.1 % (13,848 positions)

Interpretation. Pooled over 19,786 paired questions (ThaiExam counted once, via the letter-based template) the quantization cost is real but under one accuracy point, and it concentrates on English reasoning tails — no Thai task degrades significantly. A Thai-fluent 12-domain qualitative review found no systematic degradation (orthography intact, no new repetition behavior); the one caution is verbatim quotation (e.g., legal statutes), where 4-bit weights can paraphrase — use retrieval grounding or BF16 for citation-critical work.

DGX Spark performance (vllm bench serve, median of 3, no speculative decoding)

Table with columns: Workload, BF16, NVFP4, Speedup
Workload	BF16	NVFP4	Speedup
1K in / 128 out, 1 stream	27.1 tok/s	63.1 tok/s	2.33×
1K in / 128 out, 4 streams	59.6 tok/s	145.4 tok/s	2.44×
128 in / 1K out, 4 streams	69.4 tok/s	174.8 tok/s	2.52×
TTFT p50 (1K prompt)	326 ms	140 ms	2.32×

Thai context vs other local models (same protocol)

Table with columns: Model, Belebele-TH, ThaiExam, Thai bits/byte ↓
Model	Belebele-TH	ThaiExam	Thai bits/byte ↓
ThaiLLM-30B NVFP4 (this repo)	0.766	0.614	0.286
ThaiLLM-30B BF16	0.770	0.619	0.273
Typhoon2.5-30B-A3B (instruct)	0.856	0.604	0.480
SEA-LION v4.5-27B (instruct)	0.843	0.619	0.591

How it was made

bash
# TensorRT Model Optimizer 0.43.0, examples/llm_ptq/hf_ptq.py
python hf_ptq.py --pyt_ckpt_path ThaiLLM/ThaiLLM-30B \
  --qformat nvfp4 --kv_cache_qformat none \
  --calib_size 512 --calib_seq 512 --batch_size 0 \
  --dataset thai_en_calib.jsonl \
  --export_path ThaiLLM-30B-NVFP4

Full reproducible pipeline (calibration builder, fairness-protocol eval scripts, custom letter-scored ThaiExam lm-eval tasks, paired statistics, complete technical report): github.com/spped2000/thaillm-nvfp4-dgx-spark

Roadmap — planned next phases

Draft-model speculative decoding: this base has no MTP head, but a same-tokenizer drafter (e.g. Qwen3-0.6B) is expected to add ~1.3–1.6× decode speed losslessly (63 → ~80–100 tok/s on DGX Spark).
Thai instruction SFT on the BF16 base → re-quantize with the same recipe → re-run the eval gate.
EAGLE-3 head training post-SFT (~1.5–2× additional).
FP8 KV cache in production (--kv-cache-dtype fp8) — disabled in this study only for measurement isolation.

Limitations

Base model: requires instruction fine-tuning before assistant use (the recommended production flow is SFT the BF16 base → re-quantize with this recipe → re-run the eval gate).
Evaluated with automatic metrics + a 12-domain expert review; no large-scale human evaluation.
Serving requires NVFP4-capable hardware (NVIDIA Blackwell) for the measured speedups.

Attribution

Base model by the ThaiLLM project (Apache-2.0). Quantization, evaluation, and release by AGIcafet, 15 July 2026.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

NatdhanaiPython

Model Tree

Base

ThaiLLM/ThaiLLM-30B

Quantized

this model

Input Modalities

Text

Output Modalities