GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
caiovicentino1
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
caiovicentino1
Model Tree
Jackrong/Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distilled-v2
Input Modalities
Output Modalities
Supported Functionality
bash
pip install vllmvllm serve caiovicentino1/Qwen3.5-9B-Claude-Opus-HLWQ-Q5 --language-model-only --enforce-eager
That's it. No plugin, no pip install polarquant, no custom code.
Tested results:
| GPU | tok/s |
|---|---|
| A100 80GB | 168 tok/s (9B) |
| RTX PRO 6000 96GB | 44 tok/s (9B) / 18 tok/s (27B) |
bash
pip install polarquant
python
import polarengine_vllm # auto-registers with transformersfrom transformers import AutoModelForCausalLM, AutoTokenizermodel = AutoModelForCausalLM.from_pretrained("caiovicentino1/Qwen3.5-9B-Claude-Opus-HLWQ-Q5", device_map="auto", trust_remote_code=True)tokenizer = AutoTokenizer.from_pretrained("caiovicentino1/Qwen3.5-9B-Claude-Opus-HLWQ-Q5", trust_remote_code=True)inputs = tokenizer("Hello!", return_tensors="pt").to("cuda")out = model.generate(**inputs, max_new_tokens=100)print(tokenizer.decode(out[0], skip_special_tokens=True))
| GPU | VRAM | Works? | Expected tok/s |
|---|---|---|---|
| RTX 4060 | 8 GB | YES | ~20 |
| RTX 3060/4070 | 12 GB | YES | ~30 |
| RTX 4080 | 16 GB | YES | ~35 |
| RTX 4090 | 24 GB | YES | ~40 |
| A100 |
Standard INT4 (GPTQ/AWQ) quantizes weights directly — outliers cause errors.
HLWQ adds a preprocessing step:
| Method | PPL (lower = better) |
|---|---|
| BF16 baseline | 6.37 |
| HLWQ → INT4 | 6.56 |
| Direct INT4 | 6.68 |
Same speed as GPTQ/AWQ, better quality.
| Flag | Why |
|---|---|
--language-model-only | Qwen3.5 is multimodal — this skips the vision encoder (we only quantized text) |
--enforce-eager | Required on Blackwell GPUs (cc 12.0). Optional on A100/H100 (faster without it) |
pip install polarquant| 80 GB |
| YES |
| ~168 |