Quantization details
- Scheme — W8A16 — INT8 symmetric weights, group size 32, BF16 activations, stored as compressed-tensors
pack-quantized.
- Kept in BF16 — the patch embedder (
vision_embedder), vision/audio projections, token embeddings (tied with lm_head), and all norms — this Unified model has no separate vision encoder, so image/audio tokens pass through the quantized decoder. These stay unquantized to preserve quality — about 9% of all parameters in this build.
- Tokenizer /
chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block. Quantized from google/gemma-4-12B-it at commit 707f0a3; Google replaced tokenizer.json on 2026-10-30 (commit 43c9d01, glitch-token fix), which this repo does not include yet.
Quantization method — PTQ (AWQ / RTN) vs QAT
This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
- RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Not used for this model's AWQ-8bit, AWQ-4bit, NVFP4A16 and NVFP4 builds, which all apply AWQ (see each repo's
recipe.yaml).
- AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
- QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.
Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN.
This build: AWQ — a calibration-based PTQ (224-sample calibration, no gradient).
Benchmarks (measured)
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, completion-style prompts without a chat template (0- or 5-shot, as listed in each row). MMLU here is lm-eval's generative variant (mmlu_generative: the model writes the answer letter and only an exact match on the first line counts), so it is not directly comparable with MMLU scores measured another way (e.g. log-likelihood scoring). Percentages are recovery vs. the BF16 original.
Table with columns: Benchmark, BF16 (original), FP8-dynamic, AWQ-8bit (this repo), NVFP4A16, NVFP4, AWQ-4bit| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit (this repo) | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|
| MMLU (5-shot, generative) | 65.40 | 65.90 (100.8%) | 65.80 (100.6%) | 67.00 (102.4%) | 62.78 (96.0%) | 63.42 (97.0%) |
| GSM8K (5-shot, strict) | 63.38 | 68.99 (108.9%) | 65.05 (102.6%) | 64.59 (101.9%) | 60.42 (95.3%) | 63.46 (100.1%) |
| ARC-Challenge (0-shot) | 42.32 | 41.64 (98.4%) | 39.59 (93.5%) | 37.63 (88.9%) | 40.96 (96.8%) | 40.61 (96.0%) |
| Winogrande (5-shot) | 61.80 | 61.56 (99.6%) | 62.75 (101.5%) | 60.14 (97.3%) | 58.72 (95.0%) | 61.17 (99.0%) |
| KMMLU (5-shot, direct) | 43.28 | 42.59 (98.4%) | 43.60 (100.7%) | 39.00 (90.1%) | 33.66 (77.8%) | 34.01 (78.6%) |
| HAE-RAE (0-shot) | 64.62 | 64.62 (100.0%) | 65.08 (100.7%) | 62.51 (96.7%) | 58.75 (90.9%) | 59.30 (91.8%) |
| Average | 56.80 | 57.55 (101.3%) | 56.98 (100.3%) | 55.15 (97.1%) | 52.55 (92.5%) | 53.66 (94.5%) |
Evaluation (measured)
Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ attention + MLP(gate/up).
Table with columns: Metric, This model, BF16 original| Metric | This model | BF16 original |
|---|
| Perplexity (lower is better) | 1.1508 | 1.1488 (+0.2%) |
| Top-1 next-token agreement | 98.8% | 99.2% |
These numbers measure fidelity to the base model, not absolute task scores.
Requirements
Table | |
|---|
| Runtime | vLLM — the architecture is in vLLM ≥ 0.23.0; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 13.7 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=5.10.1 + compressed-tensors>=0.15.0 for text. Image input under Transformers is broken for the AWQ / NVFP4 builds (their ignore list misses the renamed vision modules). |
| trust_remote_code | not needed |
Usage (vLLM)
vllm serve axteam/gemma-4-12B-it-AWQ-8bit
from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-12B-it-AWQ-8bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)
Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64.
Thinking mode is off by default; turn it on with chat_template_kwargs={"enable_thinking": True} (Python llm.chat(..., chat_template_kwargs=...) or the request body), and add --reasoning-parser gemma4 to vllm serve to return the reasoning separately.
Reducing KV-cache memory (optional, serve-time)
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints
(RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice,
independent of the weight quantization above. For long context or large batches you can
roughly halve KV-cache memory with FP8:
vllm serve axteam/gemma-4-12B-it-AWQ-8bit --kv-cache-dtype fp8
--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2.
For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.
License
This is a quantized derivative of Google's Gemma 4. Use is governed by the
Gemma 4 license. All original
weights, tokenizer, and chat template belong to Google DeepMind.
한국어 요약
google/gemma-4-12B-it의 AWQ INT8 (W8A16) 양자화 빌드 · 제작 axteam
개요
Table with columns: 항목, 내용| 항목 | 내용 |
|---|
| 원본 모델 | google/gemma-4-12B-it |
| 양자화 방식 | AWQ INT8 (W8A16) |
| 저장 형식 | compressed-tensors pack-quantized |
| 모델 구조 | Gemma 4 · Dense |
| 용량 | 약 13.7 GB |
| 보정 데이터 | 샘플 224개 (한국어 위주, thinking 모드 답변 없음) |
양자화 상세
- 가중치: INT8 대칭, 그룹 크기 32
- 활성화: BF16 (양자화하지 않음)
- 양자화 절차: PTQ(학습 후 양자화, 재학습 없음), AWQ 방식 — 보정 데이터로 활성화가 큰 가중치 채널을 찾아 채널별 스케일로 보호한 뒤 INT8로 반올림
- AWQ 적용 범위: 어텐션 입력·MLP(gate/up)
- BF16 유지: 패치 임베더(
vision_embedder), 비전·오디오 투영층, 토큰 임베딩(LM 헤드와 공유), 모든 정규화층 — 전체 파라미터의 약 9%
- 양자화 대상: 디코더의 어텐션·MLP 선형층(projection), 별도 비전 인코더가 없는 Unified 구조라 이미지·오디오 토큰도 양자화된 디코더를 거침
- 원본 파일: 토크나이저·chat template(Google의 2026년 7월 수정본)·processor·generation config를 바이트 단위로 그대로 사용,
config.json은 원본에 quantization_config 블록만 추가
- 원본 커밋:
google/gemma-4-12B-it 커밋 707f0a3 기준 — Google이 2026-10-30에 교체한 tokenizer.json(커밋 43c9d01, glitch token 수정)은 아직 미반영
품질 (실측)
- 원본 대비 충실도: PPL 1.1508 (원본 1.1488, +0.2%, 낮을수록 원본에 가까움), 다음 토큰 top-1 일치율 98.8% (원본 자체 기준값 99.2%) — 내부 영어·한국어 혼합 평가셋, 원본(BF16)과 비교 측정
- 벤치마크 평균: 56.98 (원본 56.80, 100.3%) — MMLU(생성형)·GSM8K·ARC-Challenge·Winogrande·KMMLU·HAE-RAE 6종 평균, lm-evaluation-harness + vLLM, chat template 없는 완성형 프롬프트(과제별 0·5-shot, 표 참고)·greedy, 모든 빌드를 같은 설정으로 측정
- 한국어 벤치마크: KMMLU 43.60 (원본 43.28, 100.7%) · HAE-RAE 65.08 (원본 64.62, 100.7%)
- MMLU 방식: lm-eval 생성형 변형(
mmlu_generative) — 모델이 답 글자를 생성하고 첫 줄이 정확히 일치해야 정답으로 채점하므로, 다른 방식(예: log-likelihood 채점)으로 잰 MMLU 점수와 직접 비교 불가
요구사항
- 실행 엔진: vLLM 0.23.0 이상(이 구조를 지원하는 첫 버전), 이 빌드는 vLLM 0.32.0에서 실행 확인
- vLLM 0.32 설치 조건: Python 3.11~3.14, torch 2.13(CUDA 13.0 빌드, CUDA 13 지원 드라이버 필요), transformers 5.16.1 이상 5.20 미만, compressed-tensors 0.17.0 — transformers·compressed-tensors는 vLLM이 버전을 고정하므로 compressed-tensors를 따로 업그레이드하지 말 것 (이 체크포인트는 0.19.0으로 저장됐지만 0.17.0에서도 로드됨)
- GPU: NVIDIA Turing(sm75) 이상, V100(sm70) 미지원 — 가중치 전용 정수 형식이라 FP8·FP4 전용 텐서코어 불필요 (GPU별 지원 여부는 vLLM 0.32 커널 선택 코드 기준, 실제 실행은 B200에서만 확인)
- 메모리: 가중치 13.7 GB + KV 캐시(컨텍스트 길이 × 동시 요청 수에 비례) + 활성화·CUDA graph용 수 GB — GPU 한 장에 다 들어가지 않으면
--tensor-parallel-size N으로 분산
- Transformers로 직접 사용: transformers 5.10.1 이상 + compressed-tensors 0.15.0 이상, 텍스트 입력만 — 이 빌드는 Transformers에서 이미지 입력 불가 (ignore 목록이 이름이 바뀐 비전 모듈을 놓침), 이미지는 vLLM 사용
- trust_remote_code: 불필요
사용법
- 서버 실행:
vllm serve axteam/gemma-4-12B-it-AWQ-8bit
- Python:
LLM(model="axteam/gemma-4-12B-it-AWQ-8bit").chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고
- 권장 샘플링:
temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)
- thinking 모드: 기본 꺼짐 — 켜려면
chat_template_kwargs={"enable_thinking": True} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": true}), 추론 과정을 답변과 분리해 받으려면 vllm serve에 --reasoning-parser gemma4 추가
- KV 캐시 절감(선택):
vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)
라이선스
- 라이선스: Apache-2.0 (Gemma 4 라이선스 안내) — 원본 모델의 라이선스를 그대로 따름
- 권리: 원본 가중치·토크나이저·chat template은 Google DeepMind 소유, 이 리포는 양자화 파생물