Quantization details
- Scheme — W4A16 — INT4 asymmetric weights, group size 32 (BF16 scale + INT4 zero-point), BF16 activations, stored as compressed-tensors
pack-quantized.
- Kept in BF16 — the patch embedder (
vision_embedder), vision/audio projections, token embeddings (tied with lm_head), and all norms — this Unified model has no separate vision encoder, so image/audio tokens pass through the quantized decoder. These stay unquantized to preserve quality — about 9% of all parameters in this build.
- Tokenizer /
chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block. Quantized from google/gemma-4-12B-it at commit 707f0a3; Google replaced tokenizer.json on 2026-10-30 (commit 43c9d01, glitch-token fix), which this repo does not include yet.
Quantization method — PTQ (AWQ / RTN) vs QAT
This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
- RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Used only as an internal baseline here, not published.
- AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
- QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.
Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN.
This build: AWQ — a calibration-based PTQ (224-sample calibration, no gradient).
Benchmarks (measured)
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
Table with columns: Benchmark, BF16 (original), FP8-dynamic, AWQ-8bit, NVFP4A16, NVFP4, AWQ-4bit (this repo)| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit | NVFP4A16 | NVFP4 | AWQ-4bit (this repo) |
|---|
| MMLU (5-shot) | 65.40 | 65.90 (100.8%) | 65.80 (100.6%) | 67.00 (102.4%) | 62.78 (96.0%) | 63.42 (97.0%) |
| GSM8K (5-shot, strict) | 63.38 | 68.99 (108.9%) | 65.05 (102.6%) | 64.59 (101.9%) | 60.42 (95.3%) | 63.46 (100.1%) |
| ARC-Challenge (0-shot) | 42.32 | 41.64 (98.4%) | 39.59 (93.5%) | 37.63 (88.9%) | 40.96 (96.8%) | 40.61 (96.0%) |
| Winogrande (5-shot) | 61.80 | 61.56 (99.6%) | 62.75 (101.5%) | 60.14 (97.3%) | 58.72 (95.0%) | 61.17 (99.0%) |
| KMMLU (5-shot, direct) | 43.28 | 42.59 (98.4%) | 43.60 (100.7%) | 39.00 (90.1%) | 33.66 (77.8%) | 34.01 (78.6%) |
| HAE-RAE (0-shot) | 64.62 | 64.62 (100.0%) | 65.08 (100.7%) | 62.51 (96.7%) | 58.75 (90.9%) | 59.30 (91.8%) |
| Average | 56.80 | 57.55 (101.3%) | 56.98 (100.3%) | 55.15 (97.1%) | 52.55 (92.5%) | 53.66 (94.5%) |
⚠️ Note: average recovery is 94.5% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.
Evaluation (measured)
Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ attention + MLP(gate/up).
Table with columns: Metric, This model, BF16 original| Metric | This model | BF16 original |
|---|
| Perplexity (lower is better) | 1.3230 | 1.1488 (+15.2%) |
| Top-1 next-token agreement | 90.0% | 99.2% |
These numbers measure fidelity to the base model, not absolute task scores.
Requirements
Table | |
|---|
| Runtime | vLLM — the architecture is in vLLM ≥ 0.23.0; this build not run here yet (other builds of this model were, with vLLM 0.32.0). |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 8.4 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=5.10.1 + compressed-tensors>=0.15.0 for text. Image input under Transformers is broken for the AWQ / NVFP4 builds (their ignore list misses the renamed vision modules). |
| trust_remote_code | not needed |
Usage (vLLM)
vllm serve axteam/gemma-4-12B-it-AWQ-4bit
from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-12B-it-AWQ-4bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)
Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64.
Thinking mode is off by default; turn it on with chat_template_kwargs={"enable_thinking": True} (Python llm.chat(..., chat_template_kwargs=...) or the request body), and add --reasoning-parser gemma4 to vllm serve to return the reasoning separately.
Reducing KV-cache memory (optional, serve-time)
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints
(RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice,
independent of the weight quantization above. For long context or large batches you can
roughly halve KV-cache memory with FP8:
vllm serve axteam/gemma-4-12B-it-AWQ-4bit --kv-cache-dtype fp8
--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2.
For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.
License
This is a quantized derivative of Google's Gemma 4. Use is governed by the
Gemma 4 license. All original
weights, tokenizer, and chat template belong to Google DeepMind.
한국어 요약
google/gemma-4-12B-it 를 AWQ INT4 (W4A16) 로 양자화한 서빙용(4-bit) 모델입니다. 작성: axteam.
- 방식 — INT4 비대칭, 32개 묶음(BF16 스케일+INT4 영점), 활성화는 BF16(W4A16). AWQ로 배율을 옮긴 뒤 양자화. 형식은 compressed-tensors
pack-quantized.
- BF16 유지 — 패치 임베더(
vision_embedder)·비전/오디오 투영·토큰 임베딩(lm_head와 묶임)·노름 — 별도 비전 인코더가 없는 Unified 구조라 이미지·오디오 토큰도 양자화된 디코더를 지남 (이 빌드 전체 파라미터의 약 9%).
- 토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로).
config.json 은 원본 + quantization_config.
- 양자화 방식(PTQ vs QAT) — 학습 후(PTQ) 방식이라 재학습 없이 원본 가중치를 저비트로 바꿉니다. RTN=보정 없이 반올림만(가장 빠르나 이상치 보상이 안 돼 4비트 품질 최저, 내부 기준용·비공개) · AWQ=소량 보정으로 큰 활성화와 곱해지는 중요 채널을 채널 스케일로 보호한 뒤 반올림(경사 없음, 4비트서 RTN보다 훨씬 좋음) · QAT=양자화를 흉내 내며 재학습해 가중치가 오차에 적응(저비트 최상이나 학습 스택·데이터·GPU 필요, 구글 Gemma QAT 체크포인트는 별도 — 본 레포는 QAT 아님). 저비트 품질 ≈ QAT ≥ AWQ > RTN. 본 빌드: AWQ — 보정 기반 PTQ(224문항 보정, 경사 없음).
- 품질(실측) — BF16 원본 대비 perplexity 1.3230(원본 1.1488, +15.2%), top-1 일치도 90.0%(원본 99.2%). 내부 영/한 평가셋에서 측정. 변형: AWQ attention + MLP(gate/up).
- 벤치마크(실측) — MMLU · GSM8K · ARC-Challenge · Winogrande · KMMLU · HAE-RAE 평균 BF16 56.80 → 이 모델 53.66 (회복률 94.5%). lm-evaluation-harness · vLLM, 모든 형식 같은 설정.
- ⚠️ 회복률 94.5% — 이 정밀도에서 열화가 눈에 띔. 품질 우선이면 8비트 권장.
- 요구사항 — vLLM ≥0.23.0; GPU Turing(sm75) 이상 — 가중치 전용 INT, 특수 텐서코어 불필요; 가중치 8.4 GB + KV 캐시; Transformers 직접 사용: transformers ≥5.10.1(텍스트). AWQ·NVFP4 빌드는 Transformers 에서 이미지 입력이 깨짐.
- 사용 —
vllm serve axteam/gemma-4-12B-it-AWQ-4bit. 권장 샘플링 temperature=1.0, top_p=0.95, top_k=64.
- KV 캐시 절감(선택, 서빙 시) — 기본은 BF16 KV(양자화 체크포인트 표준, RedHat·NVIDIA 동일). 가중치 양자화와 무관한 런타임 선택. 긴 문맥·대배치에선
vllm serve … --kv-cache-dtype fp8 로 KV 메모리 절반. (GGUF라면 llama.cpp --cache-type-k q8_0 --cache-type-v q8_0)
- 라이선스 — 구글 Gemma 4 파생물. Gemma 4 라이선스 적용. 원본 가중치·토크나이저·chat_template 권리는 Google DeepMind.