GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
mattbucci
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
mattbucci
Model Tree
Qwen/Qwen3.5-27B
Input Modalities
Output Modalities
Supported Functionality
| Checkpoint | basic ("capital of France?") | thinking |
|---|---|---|
| v1 of this repo (Open-Platypus calibration) and most community AWQ | ❌ empty content (model loops in <think> until max_tokens) | ❌ |
| v2 (current) | ✅ "Paris" with finish_reason=stop, 45 reasoning tokens | ✅ engages thinking, terminates cleanly on simple QA |
The default AWQ calibration recipes (Open-Platypus, ShareGPT, etc.) have no <think> traces in the assistant turns. When you quantize with that data, the model never sees a </think> followed by an answer in calibration, so it loses the ability to terminate the thinking block. Result: validate_capabilities.py basic test ("What is the capital of France? Answer in one word.") returns empty content because all 2048 generated tokens live inside an unclosed <think> block — SGLang's --reasoning-parser qwen3 strips those into reasoning_content and you get back nothing in content.
This checkpoint was calibrated with a thinking-aware mixed dataset:
<think>...</think> traceswith tokenizer.apply_chat_template(..., enable_thinking=True) so the <think>...</think> structure appears in every render. 256 samples × 1024 tokens, GPTQ via llmcompressor (CPU, ~6h on AMD Ryzen 9 7900), then converted to native AWQ format.
Do NOT use temperature=0 (greedy decode) — Qwen3-family models loop on greedy: "Paris\n</think>\nParis\n</think>...". Use the model's recommended sampling, which SGLang picks up automatically via sampling_defaults='model':
markdown
temperature=0.7top_p=0.95top_k=20
Validator confirms temperature=0.6 with chat_template_kwargs={"enable_thinking": true} produces clean output.
validate_capabilities.py --skip-vision --skip-video (from the calibration repo):
markdown
[PASS] basic finish=stop answer='paris' (45 reasoning tokens, was BROKEN on original)[~] thinking reasoning_seen answer_ok (model derives correct $0.05 for theball-and-bat puzzle in ~400 reasoningtokens at temp=0.6; verbose at temp=0.7— bumped validator budget to 4096 tok)
Thinking on simple QA terminates in tens of tokens. Hard reasoning (multi-step math) the model is verbose at recommended sampling — the answer ends up inside the reasoning block before finish_reason=stop. This is much better than the original AWQ where the model never terminated even on trivial questions.
Qwen3_5ForConditionalGeneration, 48 layers, hybrid DeltaNet + full-attention)lm_head, DeltaNet in_proj_a/in_proj_b (recurrent state — INT4 destroys it), vision towerqweight + scales + qzeros). Loadable by SGLang's AWQ Triton + HIP GEMV kernels.model-vision.safetensors carries the BF16 vision tower + preprocessor_config.json is included so multimodal inference still works.| File | What it is |
|---|---|
model.safetensors (+ index.json) | AWQ language-model weights |
model-vision.safetensors | BF16 vision tower (untouched, preserved from base) |
chat_template.jinja | Original Qwen3.5 template (supports enable_thinking) |
tokenizer.json, tokenizer_config.json | Stock Qwen3.5 tokenizer |
| , |
bash
MODEL=mattbucci/Qwen3.5-27B-AWQ-4bit-calibrated python -m sglang.launch_server \--model-path $MODEL \--tensor-parallel-size 2 \--dtype float16 \--kv-cache-dtype fp8_e4m3 \--context-length 262144 \--quantization awq \--reasoning-parser qwen3 \--disable-cuda-graph \--disable-custom-all-reduce \--disable-overlap-schedule \--attention-backend triton
Or via the reference launcher: MODEL=$MODEL ./scripts/launch.sh qwen35
Single-user, FP8 KV cache, --disable-cuda-graph:
| Context | tok/s | TTFT |
|---|---|---|
| 128 | 26 | 5s |
| 16K | 18.7 | 8.7s |
| 32K | 15.3 | 20s |
| 65K | 13.0 | 47s |
| 131K | 9.5 | 100s |
| 256K | 5.8 | 209s |
Dense DeltaNet hybrid — bandwidth-bound at short context, full-attention layers dominate at long. For 256K agent workloads, prefer Qwen3.6-35B-A3B MoE (12-13 tok/s @ 256K).
All scripts, validator, and benchmark code: github.com/mattbucci/2x-R9700-RDNA4-GFX1201-sglang-inference — see scripts/quantize/quantize_qwen35_thinking_aware.py and scripts/quantize/calibration_datasets.py.
Apache 2.0, inherited from the base model.
preprocessor_config.jsonprocessor_config.json| Multimodal processor configs |
config.json, generation_config.json | Stock + AWQ quantization config |