Model Details
Table with columns: Property, Value| Property | Value |
|---|
| Base model | google/gemma-4-12B-it (BF16, ~24.4 GB) |
| Quantization format | MXFP4 W4A16 (compressed-tensors) |
| Disk size | 7.9 GB (68% reduction from BF16) |
| VRAM (model weights) | ~8 GB |
| Quantized scope | Decoder Linear layers (lm_head, embed_vision, embed_audio kept in BF16) |
| Calibration data | BCCard on-policy dataset (Korean/English split) |
The 12B checkpoint is the gemma4_unified architecture variant — its vision
stack (embed_vision.*) differs from the 31B's vision_tower, and is kept
at original precision.
Serving
vLLM (recommended)
# Standard environment (CUDA 12.9+):
vllm serve gemma-4-12B-it-MXFP4 \
--port 8000 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
# Blackwell with CUDA < 12.9:
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve gemma-4-12B-it-MXFP4 \
--port 8000 \
--linear-backend marlin \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
Blackwell Compatibility
On Blackwell GPUs (SM 12.x) with containers running CUDA < 12.9, FlashInfer JIT compilation fails. The following workarounds are required:
Table with columns: Setting, Purpose| Setting | Purpose |
|---|
VLLM_USE_FLASHINFER_SAMPLER=0 | Disables FlashInfer sampler (uses native PyTorch) |
--linear-backend marlin | Uses pre-compiled MXFP4 GEMM kernels (no JIT) |
Component matrix after workarounds:
Table with columns: Component, Backend, JIT Required| Component | Backend | JIT Required |
|---|
| MXFP4 GEMM | Marlin (pre-compiled) | No |
| Attention | TRITON_ATTN | No (Triton JIT, not nvcc) |
| Sampling | Native PyTorch | No |
Evaluation
Quality Benchmarks
Table with columns: Task, MXFP4 (this model), BF16 baseline, Recovery| Task | MXFP4 (this model) | BF16 baseline | Recovery |
|---|
| HellaSwag (English) | 83.5% | 81.0% | 103.1% |
| KoBEST-HellaSwag | 62.0% | 64.0% | 96.9% |
| KoBEST-COPA | 98.5% | 98.5% | 100.0% |
| KoBEST-BoolQ | 94.0% | 93.0% | 101.1% |
Korean Knowledge (KMMLU) — Reference Points
Published scores from other Korean(-capable) models, for orientation.
These are NOT apples-to-apples: vendors use the full 45-subject KMMLU
(or the harder KMMLU-Redux subset) with their own prompting/shot settings,
while this model's score is a 5-subject, 200-sample, 0-shot chat-MCQ sample
from the harness above.
Table with columns: Model, Params, Disk, Score, Benchmark (setting/source)| Model | Params | Disk | Score | Benchmark (setting/source) |
|---|
| A.X 4.0 (SKT) | 72B | ~144 GB | 78.3 | KMMLU, vendor |
| EXAONE 4.0 (LG) | 32B | ~64 GB | 75.2 | KMMLU, as reported in the HyperCLOVA X 32B Think report |
| GPT-4o (OpenAI) | — |
Disk sizes for third-party models are approximate BF16 weight footprints
(params × 2 bytes); this repo's model and its BF16 baseline are measured
on-disk values.
Directional takeaway: at 12B/4-bit this model lands well above the previous
Gemma generation's 12B (and its 27B on the harder Redux subset) on Korean
knowledge, while purpose-built Korean 32B+/72B models remain ahead — as
expected for their size. Against its own BF16 baseline under the identical
harness, the model averages 99.7% recovery across all scored tasks,
with KMMLU (91.7%) the only drop beyond sampling noise.
Compression Comparison
Table with columns: Format, Disk Size, vs BF16, Notes| Format | Disk Size | vs BF16 | Notes |
|---|
| BF16 (original) | ~24.4 GB | 100% | google/gemma-4-12B-it |
| FP8-dynamic | 13 GB | ~53% | community quantization with the same llm-compressor FP8_DYNAMIC recipe (no official RedHatAI 12B release) |
| This Model (MXFP4) | 7.9 GB | |
Citation
@misc{gemma4-12b-mxfp4-mhr-psa,
title={Gemma 4 12B IT MXFP4},
author={Taeyoung Lee},
year={2026},
url={https://huggingface.co/BCCard/MoAI-gemma-4-12B-it-mxfp4}
}
License
This model is licensed under the Apache 2.0.
