GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
zai-org/GLM-4.7-Flash
Input Modalities
Output Modalities
Supported Functionality
nvfp4-pack-quantized.mlp.gate, incl. e_score_correction_bias) is kept in BF16. The MTP layer (model.layers.47: eh_proj, shared_head.head) is quantized like the other layers.chat_template identical to the original.q_a_proj + kv_a_proj_with_mqa (MLA) and gate_proj + up_proj of every routed and shared expert.Revision 2026-10-31: the first upload of this repo (same day) quantized those pairs with separate global scales. vLLM runs each pair as one GEMM with one scale, so that build produced corrupted text. The weights were re-quantized and replaced; if you downloaded this repo before the fix, please download it again.
Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | NVFP4A16 (this repo) |
|---|---|---|---|
| MMLU (5-shot) | 72.19 | 71.78 (99.4%) | 69.97 (96.9%) |
| GSM8K (5-shot, strict) | 83.70 | 82.71 (98.8%) | 79.83 (95.4%) |
| ARC-Challenge (0-shot) | 57.51 | 57.51 (100.0%) | 55.03 (95.7%) |
| Winogrande (5-shot) | 74.51 | 73.32 (98.4%) | 73.40 (98.5%) |
| KMMLU (5-shot, direct) | 50.04 | 50.15 (100.2%) | 47.38 (94.7%) |
| HAE-RAE (0-shot) | 64.89 | 63.89 (98.4%) | 59.67 (91.9%) |
| Average | 67.14 | 66.56 (99.1%) | 64.21 (95.6%) |
⚠️ Note: average recovery is 95.6% — noticeable degradation at this precision; prefer an 8-bit build if quality matters most.
| Runtime | vLLM — the architecture is in vLLM ≥ 0.15.0; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only FP4, no FP4 tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 18.9 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | Not loadable correctly by Transformers (checked up to 5.22.1) — use vLLM. |
| trust_remote_code | not needed |
Start an OpenAI-compatible server:
bash
vllm serve axteam/GLM-4.7-Flash-NVFP4A16 --max-model-len 32768Query it:
bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/GLM-4.7-Flash-NVFP4A16", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 1.0, "max_tokens": 512}'Offline inference in Python:
python
from vllm import LLM, SamplingParams llm = LLM(model="axteam/GLM-4.7-Flash-NVFP4A16", max_model_len=32768)params = SamplingParams(temperature=1.0, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)Recommended sampling (from the original
generation_config.json):temperature=1.0. Thinking mode is on by default in the chat template; to turn it off passchat_template_kwargs={"enable_thinking": False}(Pythonllm.chat(..., chat_template_kwargs=...), or"chat_template_kwargs": {"enable_thinking": false}in the request body).
The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:
bash
vllm serve axteam/GLM-4.7-Flash-NVFP4A16 --kv-cache-dtype fp8Quantized derivative of zai-org/GLM-4.7-Flash (mit).
zai-org/GLM-4.7-Flash의 NVFP4A16 (W4A16) 양자화 빌드 · 제작 axteam
| 항목 | 내용 |
|---|---|
| 원본 모델 | zai-org/GLM-4.7-Flash |
| 양자화 방식 | NVFP4A16 (W4A16, FP4 가중치) |
| 저장 형식 | compressed-tensors nvfp4-pack-quantized |
| 모델 구조 | MoE (GLM-4.7 Lite, Glm4MoeLite) · 31.2B · 컨텍스트 202K · 텍스트 전용 |
| 용량 | 약 18.9 GB |
| 보정 데이터 | 없음 (가중치만으로 스케일 계산, 이름 기반 model_free_ptq) |
mlp.gate, e_score_correction_bias 포함)model.layers.47도 다른 층과 같이 양자화)q_a_proj+kv_a_proj_with_mqa, 모든 라우팅 대상 전문가(routed expert)·공유 전문가의 gate_proj+up_proj)은 FP4 전역 스케일 하나를 공유--tensor-parallel-size N으로 분산vllm serve axteam/GLM-4.7-Flash-NVFP4A16 --max-model-len 32768 — OpenAI 호환 APILLM(model="axteam/GLM-4.7-Flash-NVFP4A16", max_model_len=32768).chat(...) — 전체 예시는 위 영어 본문 'Usage (vLLM)' 절 참고temperature=1.0 (원본 generation_config.json 기준)chat_template_kwargs={"enable_thinking": False} 전달(Python은 llm.chat() 인자, HTTP는 요청 본문의 "chat_template_kwargs": {"enable_thinking": false})vllm serve에 --kv-cache-dtype fp8 추가 — 기본 BF16인 KV 캐시를 FP8로 저장해 KV 캐시 메모리를 약 절반으로 줄임 (긴 컨텍스트·동시 요청이 많을 때 유리, 가중치 양자화와 별개인 실행 옵션)curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/GLM-4.7-Flash-NVFP4A16", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 1.0, "max_tokens": 512}'from vllm import LLM, SamplingParams llm = LLM(model="axteam/GLM-4.7-Flash-NVFP4A16", max_model_len=32768)params = SamplingParams(temperature=1.0, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)