GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
chris320211
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
chris320211
Model Tree
microsoft/Phi-3.5-mini-instruct
Input Modalities
Output Modalities
Supported Functionality
Measured on NVIDIA GeForce RTX 3090 (vast-rtx3090) against the original fp16 snapshot.
Numbers copied from jobs/20261102T212454Z-0531e9/benchmark.json.
| Quantized | fp16 snapshot | |
|---|---|---|
| Perplexity | 6.822674 | 6.625631 |
| NLL loss | 1.920252 | 1.890946 |
| Tokens/s (2048 prefill, after warmup) | 12268.5 | 4218.8 |
| Peak VRAM (GB) | 5.908 | 9.681 |
ppl_ratio: 1.029740 (quality_ok=True)improved_throughput: Trueimproved_vram: Truepython
from huggingface_hub import snapshot_downloadpath = snapshot_download("chris320211/smoothquant-phi35-mini-vastrtx3090")This is not a drop-in AutoModelForCausalLM.from_pretrained checkpoint.
Reload with the included quant_agent_inference_adapter.py after the method
repo (and overlay, if any) is on QUANT_AGENT_METHOD_REPO. See
quantization_config.json.
A native vLLM build of these weights: chris320211/smoothquant-phi35-mini-vastrtx3090-vllm.
bash
vllm serve chris320211/smoothquant-phi35-mini-vastrtx3090-vllmvLLM PPL ratio 1.0290 against vLLM fp16; prefill 2.05×, decode 1.62× (batch 1), 1.49× (batch 16). Measured on NVIDIA GeForce RTX 3090 with vLLM 0.30.0.
Inherits the license of microsoft/Phi-3.5-mini-instruct
(mit), plus the method repository license. Keep LICENSE /
NOTICE.md from the snapshot when present.