GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
google/gemma-4-E2B-it
Input Modalities
Output Modalities
Supported Functionality
pack-quantized.embed_tokens_per_layer), and all norms. These stay unquantized to preserve quality — about 64% of all parameters in this build.chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (128-sample calibration, no gradient).
Fidelity to the BF16 original on an internal mixed English/Korean set (100 prompts: the BF16 original's greedy answers, scored under each build with the same prompts and scoring code). This build was measured on the rebuilt files with vLLM 0.32 on 2026-11-01; the BF16 reference column was measured on 2026-10-12.
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 1.3327 | 1.1833 (+12.6%) |
| Top-1 next-token agreement | 89.8% | 99.0% |
These measure fidelity to the base model, not absolute task scores.
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit | NVFP4A16 | NVFP4 | AWQ-4bit (this repo) |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 55.97 | 55.49 (99.1%) | 56.25 (100.5%) | 53.40 (95.4%) | 50.06 (89.4%) | 54.57 (97.5%) |
| GSM8K (5-shot, strict) | 51.63 | 48.82 (94.6%) | 53.98 (104.6%) | 48.52 (94.0%) | 38.13 (73.9%) | 40.03 (77.5%) |
| ARC-Challenge (0-shot) | 49.57 | 48.04 (96.9%) | 49.23 (99.3%) | 48.63 (98.1%) | 45.48 (91.7%) | 44.80 (90.4%) |
| Winogrande (5-shot) | 62.59 | 60.77 (97.1%) | 62.51 (99.9%) | 61.25 (97.9%) | 58.88 (94.1%) | 60.54 (96.7%) |
| KMMLU (5-shot, direct) | 40.72 | 40.70 (100.0%) | 40.69 (99.9%) | 39.51 (97.0%) | 37.95 (93.2%) | 39.45 (96.9%) |
| HAE-RAE (0-shot) | 62.14 | 62.05 (99.9%) | 62.24 (100.1%) | 59.67 (96.0%) | 57.01 (91.7%) | 60.59 (97.5%) |
| Average | 53.77 | 52.65 (97.9%) | 54.15 (100.7%) | 51.83 (96.4%) | 47.92 (89.1%) | 50.00 (93.0%) |
⚠️ Note: average recovery is 93.0% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.
| Runtime | vLLM — the architecture is in vLLM ≥ 0.19.0; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 7.6 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=5.5.2 (5.5.0/5.5.1 cannot run the KV-shared layers). |
| trust_remote_code | not needed |
bash
vllm serve axteam/gemma-4-E2B-it-AWQ-4bitpython
from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-E2B-it-AWQ-4bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)Recommended sampling (per Google):
temperature=1.0,top_p=0.95,top_k=64. Thinking mode: add the<|think|>token at the start of the system prompt to enable step-by-step reasoning.
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:
bash
vllm serve axteam/gemma-4-E2B-it-AWQ-4bit --kv-cache-dtype fp8
--kv-cache-dtypeacceptsauto(default, BF16),fp8/fp8_e4m3,fp8_e5m2. For a GGUF build the llama.cpp equivalent is--cache-type-k q8_0 --cache-type-v q8_0.
This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.
google/gemma-4-E2B-it의 AWQ INT4 (W4A16) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | google/gemma-4-E2B-it |
| 양자화 방식 | AWQ INT4 (W4A16) |
| 저장 형식 | compressed-tensors pack-quantized |
| 모델 구조 | Gemma 4 · Dense |
| 용량 | 약 7.6 GB |
| 보정 데이터 | 샘플 128개 (한국어 위주, thinking 모드 답변 없음) |
embed_tokens_per_layer), 모든 정규화층 — 전체 파라미터의 약 64%config.json은 원본에 quantization_config 블록만 추가--tensor-parallel-size N으로 분산vllm serve axteam/gemma-4-E2B-it-AWQ-4bitLLM(model="axteam/gemma-4-E2B-it-AWQ-4bit").chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)<|think|> 토큰 추가vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)a412d9b) 이전에 받았다면 다시 받을 것, 품질 항목의 벤치마크·충실도는 재양자화 빌드로 잰 값from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-E2B-it-AWQ-4bit")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)