GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
google/gemma-4-26B-A4B-it
Input Modalities
Output Modalities
Supported Functionality
pack-quantized.chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (256-sample calibration, no gradient).
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit (this repo) | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 40.23 | 35.77 (88.9%) | 38.72 (96.2%) | 30.44 (75.7%) | 31.39 (78.0%) | 33.21 (82.5%) |
| GSM8K (5-shot, strict) | 66.19 | 67.70 (102.3%) | 67.10 (101.4%) | 50.42 (76.2%) | 60.88 (92.0%) | 55.12 (83.3%) |
| ARC-Challenge (0-shot) | 45.39 | 45.82 (100.9%) | 44.88 (98.9%) | 44.62 (98.3%) | 44.28 (97.6%) | 44.62 (98.3%) |
| Winogrande (5-shot) | 58.48 | 60.22 (103.0%) | 59.91 (102.4%) | 58.64 (100.3%) | 57.77 (98.8%) | 58.48 (100.0%) |
| KMMLU (5-shot, direct) | 36.52 | 34.81 (95.3%) | 35.84 (98.1%) | 33.42 (91.5%) | 34.37 (94.1%) | 33.91 (92.9%) |
| HAE-RAE (0-shot) | 76.17 | 75.62 (99.3%) | 75.99 (99.8%) | 73.60 (96.6%) | 72.87 (95.7%) | 70.76 (92.9%) |
| Average | 53.83 | 53.32 (99.1%) | 53.74 (99.8%) | 48.53 (90.1%) | 50.26 (93.4%) | 49.35 (91.7%) |
Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ attention.
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower is better) | 1.1296 | 1.1284 (+0.1%) |
| Top-1 next-token agreement | 98.5% | 98.7% |
These numbers measure fidelity to the base model, not absolute task scores.
| Runtime | vLLM — the architecture is in vLLM ≥ 0.19.0; this build was run here only for the evaluation above, not yet with vLLM 0.32.0 (other builds of this model were, with vLLM 0.32.0). |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; this build was run here only for the evaluation above.) |
| Memory | Weights: 29.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | Not loadable by Transformers (checked up to 5.22.1: no conversion rule for per-expert checkpoints, the experts stay uninitialised) — use vLLM. |
| trust_remote_code | not needed |
bash
vllm serve axteam/gemma-4-26B-A4B-it-AWQ-8bitpython
from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-26B-A4B-it-AWQ-8bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)Recommended sampling (per Google):
temperature=1.0,top_p=0.95,top_k=64. Thinking mode is off by default; turn it on withchat_template_kwargs={"enable_thinking": True}(Pythonllm.chat(..., chat_template_kwargs=...)or the request body), and add--reasoning-parser gemma4tovllm serveto return the reasoning separately.
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:
bash
vllm serve axteam/gemma-4-26B-A4B-it-AWQ-8bit --kv-cache-dtype fp8
--kv-cache-dtypeacceptsauto(default, BF16),fp8/fp8_e4m3,fp8_e5m2. For a GGUF build the llama.cpp equivalent is--cache-type-k q8_0 --cache-type-v q8_0.
This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.
google/gemma-4-26B-A4B-it의 AWQ INT8 (W8A16) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | google/gemma-4-26B-A4B-it (커밋 4d7ae49) |
| 양자화 방식 | AWQ INT8 (W8A16) |
| 저장 형식 | compressed-tensors pack-quantized |
| 모델 구조 | Gemma 4 · MoE (전문가 128개 중 8개 활성 + 항상 쓰는 공유 MLP) |
| 용량 | 약 29.2 GB |
| 보정 데이터 | 샘플 256개 (한국어 위주, 원본 모델이 스스로 생성한 thinking 모드 답변 32개 포함) |
config.json은 원본에 quantization_config 블록만 추가--tensor-parallel-size N으로 분산vllm serve axteam/gemma-4-26B-A4B-it-AWQ-8bitLLM(model="axteam/gemma-4-26B-A4B-it-AWQ-8bit").chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)chat_template_kwargs={"enable_thinking": True} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": true}), 추론 과정을 답변과 분리해 받으려면 vllm serve에 --reasoning-parser gemma4 추가vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-26B-A4B-it-AWQ-8bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)