GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
google/gemma-4-26B-A4B
Input Modalities
Output Modalities
Supported Functionality
pack-quantized.processor_config / generation_config are byte-identical to the original. This is a base (pre-trained) model with no chat template — use plain text completion, not chat. config.json is the original plus a quantization_config block.This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (256-sample calibration, no gradient).
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit (this repo) | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 77.77 | 77.25 (99.3%) | 77.75 (100.0%) | 77.18 (99.2%) | 76.20 (98.0%) | 76.82 (98.8%) |
| GSM8K (5-shot, strict) | 70.89 | 71.34 (100.6%) | 71.49 (100.9%) | 69.98 (98.7%) | 67.17 (94.8%) | 69.98 (98.7%) |
| ARC-Challenge (0-shot) | 68.34 | 68.60 (100.4%) | 68.17 (99.8%) | 68.52 (100.2%) | 66.98 (98.0%) | 68.69 (100.5%) |
| Winogrande (5-shot) | 80.58 | 80.03 (99.3%) | 80.58 (100.0%) | 79.64 (98.8%) | 78.22 (97.1%) | 79.56 (98.7%) |
| KMMLU (5-shot, direct) | 60.03 | 60.07 (100.1%) | 60.15 (100.2%) | 59.41 (99.0%) | 58.14 (96.9%) | 58.99 (98.3%) |
| HAE-RAE (0-shot) | 77.27 | 76.99 (99.6%) | 76.90 (99.5%) | 77.18 (99.9%) | 75.44 (97.6%) | 75.80 (98.1%) |
| Average | 72.48 | 72.38 (99.9%) | 72.51 (100.0%) | 71.98 (99.3%) | 70.36 (97.1%) | 71.64 (98.8%) |
Perplexity was measured against the BF16 original on public text — the WikiText-2 test set (English) and a Korean Wikipedia sample, 2,048-token chunks. 8-bit builds (AWQ INT8, FP8) stayed within 0.5% of BF16; 4-bit builds (AWQ INT4, NVFP4) were 2.0–6.6% higher (range across both base models and both 4-bit formats). Next-token top-1 agreement was not measured for the base models.
| Runtime | vLLM — the architecture is in vLLM ≥ 0.19.0; this build was run here only for the evaluation above, not yet with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; this build was run here only for the evaluation above.) |
| Memory | Weights: 29.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | Not loadable by Transformers (checked up to 5.22.1: no conversion rule for per-expert checkpoints, the experts stay uninitialised) — use vLLM. |
| trust_remote_code | not needed |
bash
vllm serve axteam/gemma-4-26B-A4B-AWQ-8bitpython
from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-26B-A4B-AWQ-8bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.generate("The three primary colors are", params)print(out[0].outputs[0].text)Recommended sampling (per Google):
temperature=1.0,top_p=0.95,top_k=64.
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:
bash
vllm serve axteam/gemma-4-26B-A4B-AWQ-8bit --kv-cache-dtype fp8
--kv-cache-dtypeacceptsauto(default, BF16),fp8/fp8_e4m3,fp8_e5m2. For a GGUF build the llama.cpp equivalent is--cache-type-k q8_0 --cache-type-v q8_0.
This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.
google/gemma-4-26B-A4B의 AWQ INT8 (W8A16) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | google/gemma-4-26B-A4B (커밋 24548b6) |
| 양자화 방식 | AWQ INT8 (W8A16) |
| 저장 형식 | compressed-tensors pack-quantized |
| 모델 구조 | Gemma 4 · MoE (전문가 128개 중 8개 활성 + 항상 쓰는 공유 MLP) |
| 용량 | 약 29.2 GB |
| 보정 데이터 | 샘플 256개 (한국어 위주) |
config.json은 원본에 quantization_config 블록만 추가--tensor-parallel-size N으로 분산vllm serve axteam/gemma-4-26B-A4B-AWQ-8bitLLM(model="axteam/gemma-4-26B-A4B-AWQ-8bit").generate(...) — 사전학습 모델이라 대화가 아닌 텍스트 이어쓰기temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-26B-A4B-AWQ-8bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.generate("The three primary colors are", params)print(out[0].outputs[0].text)