GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
google/gemma-4-E4B-it
Input Modalities
Output Modalities
Supported Functionality
float-quantized (scheme FP8_DYNAMIC).embed_tokens_per_layer), and all norms. These stay unquantized to preserve quality — about 51% of all parameters in this build.chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining. How close each build stays to BF16 is reported in the measured sections where available.
Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: FP8 dynamic — a calibration-free PTQ (static per-channel weight scales + runtime per-token activation scales).
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic (this repo) | AWQ-8bit | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 62.16 | 63.20 (101.7%) | 62.40 (100.4%) | 64.48 (103.7%) | 62.14 (100.0%) | 64.16 (103.2%) |
| GSM8K (5-shot, strict) | 84.31 | 84.00 (99.6%) | 84.08 (99.7%) | 82.71 (98.1%) | 79.83 (94.7%) | 81.73 (96.9%) |
| ARC-Challenge (0-shot) | 56.91 | 57.34 (100.7%) | 56.66 (99.6%) | 56.66 (99.6%) | 55.20 (97.0%) | 55.97 (98.4%) |
| Winogrande (5-shot) | 66.30 | 65.75 (99.2%) | 66.85 (100.8%) | 65.51 (98.8%) | 66.46 (100.2%) | 65.43 (98.7%) |
| KMMLU (5-shot, direct) | 51.46 | 51.24 (99.6%) | 51.46 (100.0%) | 50.15 (97.5%) | 49.11 (95.4%) | 51.31 (99.7%) |
| HAE-RAE (0-shot) | 69.84 | 69.02 (98.8%) | 69.57 (99.6%) | 69.48 (99.5%) | 68.10 (97.5%) | 68.10 (97.5%) |
| Average | 65.16 | 65.09 (99.9%) | 65.17 (100.0%) | 64.83 (99.5%) | 63.47 (97.4%) | 64.45 (98.9%) |
Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: FP8 dynamic (W8A8).
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower is better) | 1.3173 | 1.3032 (+1.1%) |
| Top-1 next-token agreement | 96.0% | 99.2% |
These numbers measure fidelity to the base model, not absolute task scores.
| Runtime | vLLM — the architecture is in vLLM ≥ 0.19.0; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 12.0 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=5.5.2 (5.5.0/5.5.1 cannot run the KV-shared layers). |
| trust_remote_code | not needed |
bash
vllm serve axteam/gemma-4-E4B-it-FP8-dynamicpython
from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-E4B-it-FP8-dynamic")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)Recommended sampling (per Google):
temperature=1.0,top_p=0.95,top_k=64. Thinking mode: add the<|think|>token at the start of the system prompt to enable step-by-step reasoning.
The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:
bash
vllm serve axteam/gemma-4-E4B-it-FP8-dynamic --kv-cache-dtype fp8
--kv-cache-dtypeacceptsauto(default, BF16),fp8/fp8_e4m3,fp8_e5m2. For a GGUF build the llama.cpp equivalent is--cache-type-k q8_0 --cache-type-v q8_0.
This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.
google/gemma-4-E4B-it의 FP8 (W8A8) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | google/gemma-4-E4B-it |
| 양자화 방식 | FP8 동적 양자화 (W8A8) |
| 저장 형식 | compressed-tensors float-quantized (scheme FP8_DYNAMIC) |
| 모델 구조 | Gemma 4 · Dense |
| 용량 | 약 12.0 GB |
| 보정 데이터 | 없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산) |
embed_tokens_per_layer), 모든 정규화층 — 전체 파라미터의 약 51%config.json은 원본에 quantization_config 블록만 추가--tensor-parallel-size N으로 분산vllm serve axteam/gemma-4-E4B-it-FP8-dynamicLLM(model="axteam/gemma-4-E4B-it-FP8-dynamic").chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고temperature=1.0, top_p=0.95, top_k=64 (Google 권장값)<|think|> 토큰 추가vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)from vllm import LLM, SamplingParams llm = LLM(model="axteam/gemma-4-E4B-it-FP8-dynamic")params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)out = llm.chat( [{"role": "user", "content": "Write a short joke about saving RAM."}], params,)print(out[0].outputs[0].text)