GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
simaai
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
simaai
Model Tree
mistralai/Mistral-7B-Instruct-v0.3
Input Modalities
Output Modalities
Supported Functionality
| Component | Method | Format | Details |
|---|---|---|---|
| Decoder Linear layers | AutoRound | symmetric INT4, G256 | 224 modules; lm_head excluded; W4A16 |
lm_head | GPTQ | symmetric INT4, G256 | static actorder; block size 128; dampening 0.01 |
| Activations | — | FP16/BF16 | not activation-quantized |
Calibration used 512 fixed-order samples from NeelNanda/pile-10k, sequence length 1024, AutoRound 200 iterations, batch size 1, and device ID 0.
Full wikitext-2-raw-v1 was evaluated through lm-eval's wikitext task with no example limit and matched settings for both checkpoints.
| Checkpoint | Word perplexity | Status |
|---|---|---|
mistralai/Mistral-7B-Instruct-v0.3 | 9.013306737007 | Full run: 2026-07-15 |
| This checkpoint | 9.359999062186 | Full run: 2026-07-15 |
| Absolute degradation | +0.346692325179 | quantized - source |
| Relative degradation | +3.846449869006% | 100 * (quantized / source - 1) |
Raw JSON evidence is retained in perplexity_results/full_wikitext/ in this workspace.
Included files: README.md, quantize.py, recipe.yaml, and versions.txt.
bash
python quantize.py --model-path /path/to/Mistral-7B-Instruct-v0.3 --output-dir /path/to/new-output-directory
The quantization job completed on an NVIDIA A100 80 GB GPU on 2026-07-15 and saved scales passed the NaN/Inf check. The compressed-checkpoint smoke test passed: 225 compressed modules including lm_head loaded and completed greedy generation.
This is a pre-LLiMa quantized Hugging Face safetensors artifact. Compile separately for Sima.ai with the appropriate LLiMa release. Quantization quality can vary by language, domain, prompt format, and context length; validate your deployment.