GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
axteam
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
axteam
Model Tree
openbmb/MiniCPM5-2B
Input Modalities
Output Modalities
Supported Functionality
pack-quantized.chat_template identical to the original.Every column was measured with the same harness and settings — lm-evaluation-harness on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.
| Benchmark | BF16 (original) | FP8-dynamic | AWQ-8bit (this repo) | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 59.74 | 59.56 (99.7%) | 59.86 (100.2%) | 56.81 (95.1%) | 53.50 (89.5%) | 56.31 (94.3%) |
| GSM8K (5-shot, strict) | 73.01 | 71.27 (97.6%) | 73.01 (100.0%) | 65.20 (89.3%) | 55.04 (75.4%) | 65.96 (90.3%) |
| ARC-Challenge (0-shot) | 43.94 | 44.20 (100.6%) | 43.86 (99.8%) | 43.43 (98.8%) | 40.10 (91.3%) | 44.88 (102.1%) |
| Winogrande (5-shot) | 64.40 | 63.22 (98.2%) | 64.40 (100.0%) | 62.75 (97.4%) | 59.04 (91.7%) | 62.12 (96.4%) |
| KMMLU (5-shot, direct) | 37.65 | 38.38 (101.9%) | 38.27 (101.7%) | 35.56 (94.4%) | 34.06 (90.5%) | 30.73 (81.6%) |
| HAE-RAE (0-shot) | 39.05 | 38.13 (97.7%) | 39.14 (100.2%) | 35.38 (90.6%) | 30.16 (77.2%) | 36.94 (94.6%) |
| Average | 52.97 | 52.46 (99.0%) | 53.09 (100.2%) | 49.85 (94.1%) | 45.32 (85.6%) | 49.49 (93.4%) |
Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts, BF16 greedy answers scored under each build).
| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 1.5988 | 1.5988 (-0.0%) |
| Top-1 next-token agreement | 98.2% | 98.9% |
| Runtime | vLLM — the architecture is in vLLM; tested here with vLLM 0.32.0. |
| vLLM install | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| GPU | NVIDIA Turing (sm75) or newer — weight-only INT, no special tensor cores needed. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| Memory | Weights: 3.2 GB. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use --tensor-parallel-size N when the weights do not fit one GPU. |
| Transformers (without vLLM) | transformers>=4.56.0 + compressed-tensors>=0.15.0 (checked 4.56.0 / 4.56.2 / 4.57.6 with FP8, AWQ-4bit, NVFP4A16). |
| trust_remote_code | not needed |
Start an OpenAI-compatible server:
bash
vllm serve axteam/MiniCPM5-2B-AWQ-8bit --max-model-len 32768Query it:
bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/MiniCPM5-2B-AWQ-8bit", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 1.0, "top_p": 0.95, "max_tokens": 512}'Offline inference in Python:
python
from vllm import LLM, SamplingParams llm = LLM(model="axteam/MiniCPM5-2B-AWQ-8bit", max_model_len=32768)params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)Recommended sampling (from the original
generation_config.json):temperature=1.0,top_p=0.95. Thinking mode is on by default in the chat template; to turn it off passchat_template_kwargs={"enable_thinking": False}(Pythonllm.chat(..., chat_template_kwargs=...), or"chat_template_kwargs": {"enable_thinking": false}in the request body).
The KV cache is kept in BF16 by default. KV-cache precision is a runtime choice, independent of the weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:
bash
vllm serve axteam/MiniCPM5-2B-AWQ-8bit --kv-cache-dtype fp8Quantized derivative of openbmb/MiniCPM5-2B (Apache-2.0). Original weights & tokenizer belong to OpenBMB.
openbmb/MiniCPM5-2B 를 AWQ INT8 (W8A16) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.
pack-quantized.vllm serve axteam/MiniCPM5-2B-AWQ-8bit --max-model-len 32768 로 OpenAI 호환 서버(포트 8000), 또는 Python LLM(model="axteam/MiniCPM5-2B-AWQ-8bit").chat(...) (위 예시). 권장 샘플링 temperature=1.0, top_p=0.95. thinking 끄기: chat_template_kwargs={"enable_thinking": False}.--kv-cache-dtype fp8.curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "axteam/MiniCPM5-2B-AWQ-8bit", "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}], "temperature": 1.0, "top_p": 0.95, "max_tokens": 512}'from vllm import LLM, SamplingParams llm = LLM(model="axteam/MiniCPM5-2B-AWQ-8bit", max_model_len=32768)params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=1024)out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)print(out[0].outputs[0].text)