GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
simaai
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
simaai
Model Tree
Qwen/Qwen2.5-7B-Instruct
Input Modalities
Output Modalities
Supported Functionality
| Component | Method | Weight format | Details |
|---|---|---|---|
| Decoder Linear layers | AutoRound | symmetric INT4, G256 | 196 targets; lm_head excluded |
lm_head | GPTQ | symmetric INT4, G256 | static act-order, block size 128, dampening 0.01 |
| Mixed-precision exceptions | — | source dtype | non-Linear parameters remain at source precision |
Calibration used HuggingFaceH4/ultrachat_200k (dataset revision not captured), train_sft[:512], rendered with the source chat template. Token IDs were deterministically concatenated and packed into 512 full 1024-token spans (no shuffle); AutoRound used 200 iterations and batch size 1.
Full wikitext-2-raw-v1 evaluation used the wikitext lm-eval task, no example limit, batch size 1, CUDA, and the same evaluator for source and quantized checkpoints on 2026-07-19.
| Checkpoint | Word perplexity | Status |
|---|---|---|
Qwen/Qwen2.5-7B-Instruct source | 10.047734 | Full run |
| This UltraChat checkpoint | 11.478585 | Full run |
| Absolute degradation | 1.430851 | Lower is better |
| Relative degradation | 14.241% | 100 * (quantized / source - 1) |
Finite-scale validation and a Transformers chat-generation smoke test passed. The full PPL result was run on an RTX 6000 after an RTX 4000 attempt OOM'd.
This directory includes the exact quantize.py, recipe.yaml, and versions.txt.
bash
python quantize.py --model-path /path/to/models--Qwen--Qwen2.5-7B-Instruct --output-dir /path/to/output
Exact Python, CUDA, Torch, Transformers, llmcompressor, AutoRound, and compressed-tensors versions are recorded in versions.txt.
This is the pre-LLiMa quantized Hugging Face artifact. Compile it separately for the target Sima.ai platform and keep compiler output separate.
Quantization quality can vary by language, domain, prompt format, context length, and deployment runtime; validate the intended workload independently.