GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
BCCard
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
BCCard
Model Tree
google/gemma-4-31B-it
Input Modalities
Output Modalities
Supported Functionality
| Property | Value |
|---|---|
| Base model | google/gemma-4-31B-it (BF16, 62.5 GB) |
| Quantization format | MXFP4 W4A16 (compressed-tensors) |
| Disk size | 19.5 GB (69% reduction from BF16) |
| VRAM (model weights) | ~19 GB |
| Calibration data | BCCard on-policy 600k |
bash
# Standard environment (CUDA 12.9+):vllm serve gemma-4-31B-it-MXFP4 \--port 8000 \--max-model-len 32768 \--gpu-memory-utilization 0.90# Blackwell with CUDA < 12.9:VLLM_USE_FLASHINFER_SAMPLER=0 \vllm serve gemma-4-31B-it-MXFP4 \--port 8000 \--linear-backend marlin \--max-model-len 32768 \--gpu-memory-utilization 0.90
On Blackwell GPUs (SM 12.x) with containers running CUDA < 12.9, FlashInfer JIT compilation fails. The following workarounds are required:
| Setting | Purpose |
|---|---|
VLLM_USE_FLASHINFER_SAMPLER=0 | Disables FlashInfer sampler (uses native PyTorch) |
--linear-backend marlin | Uses pre-compiled MXFP4 GEMM kernels (no JIT) |
Component matrix after workarounds:
| Component | Backend | JIT Required |
|---|---|---|
| MXFP4 GEMM | Marlin (pre-compiled) | No |
| Attention | TRITON_ATTN | No (Triton JIT, not nvcc) |
| Sampling | Native PyTorch | No |
| Task | MXFP4 (this model) | BF16 baseline | Recovery |
|---|---|---|---|
| HellaSwag (English) | 89.5% | 91.0% | 98.4% |
| KoBEST-HellaSwag | 73.0% | 76.5% | 95.4% |
| KoBEST-COPA | 99.0% | 99.0% | 100.0% |
| KoBEST-BoolQ | 95.0% | 96.5% | 98.4% |
Published scores from other Korean(-capable) models, for orientation. These are NOT apples-to-apples: vendors use the full 45-subject KMMLU (or the harder KMMLU-Redux subset) with their own prompting/shot settings, while this model's score is a 5-subject, 200-sample, 0-shot chat-MCQ sample from the harness above.
| Model | Params | Disk | Score | Benchmark (setting/source) |
|---|---|---|---|---|
| A.X 4.0 (SKT) | 72B | ~144 GB | 78.3 | KMMLU, vendor |
| Gemma 4 31B IT (BF16 baseline) | 31B | 62.5 GB | 78.0 | KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| This model (Gemma 4 31B MXFP4, 4-bit) | 31B |
Disk sizes for third-party models are approximate BF16 weight footprints (params × 2 bytes); this repo's model and its BF16 baseline are measured on-disk values.
Directional takeaway: on this 5-subject sample the BF16 gemma-4-31B-it baseline (78.0) sits on par with A.X 4.0 72B (78.3), and the 4-bit MXFP4 model retains that tier — landing above EXAONE 4.0 32B and GPT-4o — at a 19.5 GB footprint. Against its own BF16 baseline under the identical harness, the model averages 98.7% recovery across all scored tasks.
Note: Perplexity cannot be measured with MXFP4 + Marlin backend due to NaN in
prompt_logprobs. Text generation and classification tasks work normally.
| Benchmark | BF16 Baseline | RedHatAI FP8 | Recovery% | This Model (MXFP4) |
|---|---|---|---|---|
| GSM8K | 96.11 | 96.22 | 100.1% | TBD |
| MMLU-Pro | 87.05 | 87.07 | 100.0% | TBD |
| IFEval | 94.58 | 94.89 | 100.3% |
RedHatAI FP8 reference: RedHatAI/gemma-4-31B-it-FP8-dynamic
| Format | Disk Size | vs BF16 | Notes |
|---|---|---|---|
| BF16 (original) | 62.5 GB | 100% | google/gemma-4-31B-it |
| FP8-dynamic | 33.3 GB | 53% | RedHatAI/gemma-4-31B-it-FP8-Dynamic — 89.6% FP8_E4M3 + 6.1% BF16 |
| This Model (MXFP4) | 19.5 GB | 31% | W4A16 GPTQ + MHR + PSA |
markdown
@misc{gemma4-mxfp4-mhr-psa,title={Gemma 4 31B IT MXFP4 with Multilingual Hessian Regularization and PSA-GPTQ},author={Taeyoung Lee},year={2026},url={https://huggingface.co/BCCard/MoAI-gemma-4-31B-it-mxfp4}}
This model is licensed under the Apache 2.0.
| KoBEST-SentiNeg | 99.0% | 99.0% | 100.0% |
| KoBEST-WiC | 98.5% | 99.0% | 99.5% |
| KMMLU (5 subjects) | 76.0% | 78.0% | 97.4% |
| BC-Finance QA (char-F1) | 0.387 | 0.386 | 100.3% |
| Average recovery | 98.7% |
| Korean Generation | 476 tok, rep 0.99, Hangul 59.2% | 482 tok, rep 0.99, Hangul 57.9% | — |
| English Generation | 424 tok, rep 0.99 | 421 tok, rep 1.00 | — |
| 76.0 |
| KMMLU 5-subject sample, 0-shot chat MCQ (this harness) |
| EXAONE 4.0 (LG) | 32B | ~64 GB | 75.2 | KMMLU, as reported in the HyperCLOVA X 32B Think report |
| GPT-4o (OpenAI) | — | — (API) | 72.5 | KMMLU, as reported by SKT A.X |
| HyperCLOVA X 32B Think (NAVER) | 32B | ~64 GB | 71.3 | KMMLU, vendor report |
| A.X 3.1 (SKT) | 34B | ~68 GB | 69.2 | KMMLU, vendor |
| A.X 3.1 Light (SKT) | 7B | ~14 GB | 61.7 | KMMLU, vendor |
| Gemma 3 27B IT (Google) | 27B | ~54 GB | 54.0 | KMMLU-Redux, KMMLU-Redux/Pro paper |