GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
zai-org/GLM-4.7-Flash
Input Modalities
Output Modalities
Supported Functionality
float-quantized (scheme FP8_DYNAMIC).mlp.gate, incl. e_score_correction_bias) is kept in BF16. The MTP layer (model.layers.47: eh_proj, shared_head.head) is quantized like the other layers.chat_template identical to the original.Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic (this repo) | NVFP4A16 |
|---|---|---|---|
| MMLU (5-shot) | 72.19 | 71.78 (99.4%) | 69.97 (96.9%) |
| GSM8K (5-shot, strict) | 83.70 | 82.71 (98.8%) | 79.83 (95.4%) |
| ARC-Challenge (0-shot) | 57.51 | 57.51 (100.0%) | 55.03 (95.7%) |
| Winogrande (5-shot) | 74.51 | 73.32 (98.4%) | 73.40 (98.5%) |
| KMMLU (5-shot, direct) | 50.04 | 50.15 (100.2%) | 47.38 (94.7%) |
| HAE-RAE (0-shot) | 64.89 | 63.89 (98.4%) | 59.67 (91.9%) |
| Average | 67.14 | 66.56 (99.1%) | 64.21 (95.6%) |
| Runtime | vLLM — the architecture is in vLLM ≥ 0.15.0; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 32.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=5.15.0. |
| trust_remote_code | not needed |
Start an OpenAI-compatible server:
bash
vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768Query it:
bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/GLM-4.7-Flash-FP8-dynamic", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 1.0, "max_tokens": 512}'Offline inference in Python:
python
from vllm import LLM, SamplingParams llm = LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic", max_model_len=32768)params = SamplingParams(temperature=1.0, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)Recommended sampling (from the original
generation_config.json):temperature=1.0. Thinking mode is on by default in the chat template; to turn it off passchat_template_kwargs={"enable_thinking": False}(Pythonllm.chat(..., chat_template_kwargs=...), or"chat_template_kwargs": {"enable_thinking": false}in the request body).
The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:
bash
vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --kv-cache-dtype fp8Quantized derivative of zai-org/GLM-4.7-Flash (mit).
zai-org/GLM-4.7-Flash의 FP8 (W8A8) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | zai-org/GLM-4.7-Flash |
| 양자화 방식 | FP8 동적 양자화 (W8A8) |
| 저장 형식 | compressed-tensors float-quantized (scheme FP8_DYNAMIC) |
| 모델 구조 | MoE (GLM-4.7 Lite, Glm4MoeLite) · 31.2B · 컨텍스트 202K · 텍스트 전용 |
| 용량 | 약 32.2 GB |
| 보정 데이터 | 없음 (가중치 스케일은 가중치만으로 계산, 활성화 스케일은 실행 시 계산) |
mlp.gate, e_score_correction_bias 포함)model.layers.47도 다른 층과 같이 양자화)--tensor-parallel-size N으로 분산vllm serve axteam/GLM-4.7-Flash-FP8-dynamic --max-model-len 32768 — OpenAI 호환 APILLM(model="axteam/GLM-4.7-Flash-FP8-dynamic", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고temperature=1.0 (원본 generation_config.json 기준)chat_template_kwargs={"enable_thinking": False} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": false})vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/GLM-4.7-Flash-FP8-dynamic", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 1.0, "max_tokens": 512}'from vllm import LLM, SamplingParams llm = LLM(model="axteam/GLM-4.7-Flash-FP8-dynamic", max_model_len=32768)params = SamplingParams(temperature=1.0, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)