GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
z-lab
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
z-lab
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
On Apple Silicon, we recommend serving ParoQuant models using oMLX. Refer to the docs for more details.
bash
# NVIDIA GPU (CUDA 12.9)pip install "paroquant[vllm]"# NVIDIA GPU (CUDA 13.0)pip install "paroquant[vllm]" "vllm==0.19.1" \--extra-index-url https://wheels.vllm.ai/0.19.1/cu130 \--extra-index-url https://download.pytorch.org/whl/cu130# Apple Siliconpip install "paroquant[mlx]"
bash
python -m paroquant.cli.chat --model z-lab/Qwen3.8-27B-PARO
For vLLM, you can directly use vllm serve to serve ParoQuant models:
bash
vllm serve z-lab/Qwen3.8-27B-PARO --port 8000
For other frameworks:
bash
python -m paroquant.cli.serve --model z-lab/Qwen3.8-27B-PARO --port 8000
For MLX, add --vlm if you wish to load the VLM components and use the model's multimodal features. For vLLM, VLM components are loaded by default and can be skipped with the server argument --language-model-only.
[!NOTE] The visual components in this checkpoint is stored in original precision, and only the language components are quantized to 4 bits; as a result, the model size is larger than a fully-quantized model. Avoid loading the VLM components if you are not using the multimodal features for the best efficiency.
[!NOTE] The following commands map the local cache directory to the container in order to persist kernel cache across runs. Remove
-v ...to disable this behavior.
bash
# Interactive chatdocker run --pull=always --rm -it --gpus all --ipc=host \-v $HOME/.cache/paroquant:/root/.cache/paroquant \ghcr.io/z-lab/paroquant:chat --model z-lab/Qwen3.8-27B-PARO# API server (port 8000)docker run --pull=always --rm -it --gpus all --ipc=host -p 8000:8000 \-v $HOME/.cache/paroquant:/root/.cache/paroquant \ghcr.io/z-lab/paroquant:serve --model z-lab/Qwen3.8-27B-PARO
bibtex
@inproceedings{liang2026paroquant,title = {{ParoQuant: Pairwise Rotation Quantization for Efficient Reasoning LLM Inference}},author = {Liang, Yesheng and Chen, Haisheng and Zhang, Zihan and Han, Song and Liu, Zhijian},booktitle = {International Conference on Learning Representations (ICLR)},year = {2026}}