GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
syedazeez
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
syedazeez
Model Tree
microsoft/Phi-3.5-mini-instruct
Input Modalities
Output Modalities
Supported Functionality
W4A16 scheme (4-bit symmetric INT weights, FP16 activations), group size 128, lm_head left unquantized.llmcompressor 0.12, output in compressed-tensors (pack-quantized) format.HuggingFaceH4/ultrachat_200k @ 2048 tokens.| Metric | Value |
|---|---|
| On-disk size | 2.27 GB (from 7.6 GB BF16, 3.35x) |
| KV-cache budget @ 4096 ctx | 4,608 tokens |
| Single-stream throughput | 68.7 tok/s |
| Batched x32 throughput | 1,039 tok/s |
| Kernel | Marlin W4A16 |
Quality spot-check (greedy): correctly computes 3^100 mod 7 = 4 with full Fermat's-little-theorem
working, and lists the first ten primes correctly. This is a spot check, not a full eval. 4-bit
quantization is lossy in general; these specific prompts held up.
This model was quantized with the same pipeline as two others and benchmarked together on one 6 GB card:
Same ~2.2 GB on disk, but servable context ranges 11x across the three (Phi is the heaviest KV footprint of the set, so it has the smallest context budget on a fixed card). Full writeup and code linked below.
bash
vllm serve syedazeez/Phi-3.5-mini-instruct-W4A16-G128 \ --max-model-len 4096 \ --gpu-memory-utilization 0.88python
from vllm import LLM, SamplingParams llm = LLM(model="syedazeez/Phi-3.5-mini-instruct-W4A16-G128", max_model_len=4096, gpu_memory_utilization=0.88)out = llm.chat( [[{"role": "user", "content": "State Fermat's little theorem and compute 3^100 mod 7."}]], SamplingParams(temperature=0, max_tokens=400),)print(out[0].outputs[0].text)On a 6 GB card, set
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Trueso CUDA-graph capture fits.
Released under the MIT License, the same as the original microsoft/Phi-3.5-mini-instruct. All credit for the model goes to Microsoft. Only the 4-bit quantization was performed here.
vllm serve syedazeez/Phi-3.5-mini-instruct-W4A16-G128 \ --max-model-len 4096 \ --gpu-memory-utilization 0.88from vllm import LLM, SamplingParams llm = LLM(model="syedazeez/Phi-3.5-mini-instruct-W4A16-G128", max_model_len=4096, gpu_memory_utilization=0.88)out = llm.chat( [[{"role": "user", "content": "State Fermat's little theorem and compute 3^100 mod 7."}]], SamplingParams(temperature=0, max_tokens=400),)print(out[0].outputs[0].text)