GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
BCCard
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
BCCard
Model Tree
google/gemma-4-12B-it
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Base model | google/gemma-4-12B-it (BF16, ~24.4 GB) |
| Quantization format | MXFP4 W4A16 (compressed-tensors) |
| Disk size | 7.9 GB (68% reduction from BF16) |
| VRAM (model weights) | ~8 GB |
| Quantized scope | Decoder Linear layers (lm_head, embed_vision, embed_audio kept in BF16) |
| Calibration data | BCCard on-policy dataset (Korean/English split) |
The 12B checkpoint is the
gemma4_unifiedarchitecture variant — its vision stack (embed_vision.*) differs from the 31B'svision_tower, and is kept at original precision.
bash
# Standard environment (CUDA 12.9+):vllm serve gemma-4-12B-it-MXFP4 \ --port 8000 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 # Blackwell with CUDA < 12.9:VLLM_USE_FLASHINFER_SAMPLER=0 \vllm serve gemma-4-12B-it-MXFP4 \ --port 8000 \ --linear-backend marlin \ --max-model-len 32768 \ --gpu-memory-utilization 0.90On Blackwell GPUs (SM 12.x) with containers running CUDA < 12.9, FlashInfer JIT compilation fails. The following workarounds are required:
| Setting | Purpose |
|---|---|
VLLM_USE_FLASHINFER_SAMPLER=0 | Disables FlashInfer sampler (uses native PyTorch) |
--linear-backend marlin | Uses pre-compiled MXFP4 GEMM kernels (no JIT) |
Component matrix after workarounds:
| Component | Backend | JIT Required |
|---|---|---|
| MXFP4 GEMM | Marlin (pre-compiled) | No |
| Attention | TRITON_ATTN | No (Triton JIT, not nvcc) |
| Sampling | Native PyTorch | No |
| Task | MXFP4 (this model) | BF16 baseline | Recovery |
|---|---|---|---|
| HellaSwag (English) | 83.5% | 81.0% | 103.1% |
| KoBEST-HellaSwag | 62.0% | 64.0% | 96.9% |
| KoBEST-COPA | 98.5% | 98.5% | 100.0% |
| KoBEST-BoolQ | 94.0% | 93.0% | 101.1% |
| KoBEST-SentiNeg | 99.0% | 99.5% | 99.5% |
| KoBEST-WiC | 82.0% | 79.0% | 103.8% |
| KMMLU (5 subjects) | 55.5% | 60.5% | 91.7% |
| BC-Finance QA (char-F1) | 0.350 | 0.344 | 101.7% |
| Average recovery | 99.7% | ||
| Korean Generation | 495 tok, rep 1.00, Hangul 58.8% | 492 tok, rep 1.00, Hangul 58.3% | — |
| English Generation | 430 tok, rep 0.99 | 432 tok, rep 0.99 | — |
Published scores from other Korean(-capable) models, for orientation. These are NOT apples-to-apples: vendors use the full 45-subject KMMLU (or the harder KMMLU-Redux subset) with their own prompting/shot settings, while this model's score is a 5-subject, 200-sample, 0-shot chat-MCQ sample from the harness above.
| Model | Params | Disk | Score | Benchmark (setting/source) |
|---|---|---|---|---|
| A.X 4.0 (SKT) | 72B | ~144 GB | 78.3 | KMMLU, vendor |
| EXAONE 4.0 (LG) | 32B | ~64 GB | 75.2 | KMMLU, as reported in the HyperCLOVA X 32B Think report |
| GPT-4o (OpenAI) | — | — (API) | 72.5 | KMMLU, as reported by SKT A.X |
| HyperCLOVA X 32B Think (NAVER) | 32B | ~64 GB | 71.3 | KMMLU, vendor report |
| A.X 3.1 (SKT) | 34B | ~68 GB | 69.2 | KMMLU, vendor |
| A.X 3.1 Light (SKT) | 7B | ~14 GB | 61.7 | KMMLU, vendor |
| Gemma 4 12B IT (BF16 baseline) | 12B | ~24.4 GB | 60.5 | KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| This model (Gemma 4 12B MXFP4) | 12B | 7.9 GB | 55.5 | KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| Gemma 3 27B IT (Google) | 27B | ~54 GB | 54.0 | KMMLU-Redux, KMMLU-Redux/Pro paper |
| HyperCLOVA X SEED (NAVER) | 3B | ~6.5 GB | 48.5 | KMMLU, vendor |
| Gemma 3 12B IT (Google) | 12B | ~24 GB | 46.7 | KMMLU-Redux, KMMLU-Redux/Pro paper |
Disk sizes for third-party models are approximate BF16 weight footprints (params × 2 bytes); this repo's model and its BF16 baseline are measured on-disk values.
Directional takeaway: at 12B/4-bit this model lands well above the previous Gemma generation's 12B (and its 27B on the harder Redux subset) on Korean knowledge, while purpose-built Korean 32B+/72B models remain ahead — as expected for their size. Against its own BF16 baseline under the identical harness, the model averages 99.7% recovery across all scored tasks, with KMMLU (91.7%) the only drop beyond sampling noise.
| Format | Disk Size | vs BF16 | Notes |
|---|---|---|---|
| BF16 (original) | ~24.4 GB | 100% | google/gemma-4-12B-it |
| FP8-dynamic | 13 GB | ~53% | community quantization with the same llm-compressor FP8_DYNAMIC recipe (no official RedHatAI 12B release) |
| This Model (MXFP4) | 7.9 GB | 32% | W4A16 GPTQ + MHR + PSA |
text
@misc{gemma4-12b-mxfp4-mhr-psa, title={Gemma 4 12B IT MXFP4}, author={Taeyoung Lee}, year={2026}, url={https://huggingface.co/BCCard/MoAI-gemma-4-12B-it-mxfp4}}This model is licensed under the Apache 2.0.
# Standard environment (CUDA 12.9+):vllm serve gemma-4-12B-it-MXFP4 \ --port 8000 \ --max-model-len 32768 \ --gpu-memory-utilization 0.90 # Blackwell with CUDA < 12.9:VLLM_USE_FLASHINFER_SAMPLER=0 \vllm serve gemma-4-12B-it-MXFP4 \ --port 8000 \ --linear-backend marlin \ --max-model-len 32768 \ --gpu-memory-utilization 0.90@misc{gemma4-12b-mxfp4-mhr-psa, title={Gemma 4 12B IT MXFP4}, author={Taeyoung Lee}, year={2026}, url={https://huggingface.co/BCCard/MoAI-gemma-4-12B-it-mxfp4}}