GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
MIRALABS
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
MIRALABS
Model Tree
Qwen/Qwen3.8-27B
Input Modalities
Output Modalities
Supported Functionality
The model can take 262,144 tokens of context. --max-model-len is how much of that you reserve in GPU memory. Set it to 262144 if you have the VRAM (we do, on two 3090s). Set it lower if you run out of memory, or if you want more than one request in flight — leftover VRAM goes to the KV cache.
bash
pip install "transformers>=5.15"vllm serve MIRALABS/Qwen3.8-27B-W4A16-AutoRound \--tensor-parallel-size 2 \--max-model-len 262144 \--kv-cache-dtype fp8 \--enable-prefix-caching \--enable-chunked-prefill \--reasoning-parser qwen3 \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--trust-remote-code
Need transformers>=5.15 for Qwen3_5Config. vLLM 0.19 still asks for <5; install 5.15 anyway.
The weights fit in 24 GB. Context does not — not at 262k. Start around 8k and raise --max-model-len until it OOMs.
bash
vllm serve MIRALABS/Qwen3.8-27B-W4A16-AutoRound \--tensor-parallel-size 1 \--max-model-len 8192 \--gpu-memory-utilization 0.95 \--kv-cache-dtype fp8 \--reasoning-parser qwen3 \--enable-auto-tool-choice \--tool-call-parser qwen3_coder \--trust-remote-code
For long context on a single 3090, use a GGUF in llama.cpp and offload what does not fit.
bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve MIRALABS/Qwen3.8-27B-W4A16-AutoRound \--max-model-len 1000000 \--hf-overrides '{"text_config":{"rope_parameters":{"mrope_interleaved":true,"mrope_section":[11,11,10],"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}}' \--trust-remote-code
Thinking is on by default (reasoning_effort=xhigh). The model writes inside <think>…</think>, then the answer. --reasoning-parser qwen3 puts the think in reasoning and the answer in content.
Give it room. A tiny max_tokens cuts it off mid-thought and you get an empty answer.
xhigh — defaultmedium / low — shorter thinkenable_thinking=false — skip thinkingQwen's sampling: thinking temperature=1.0, top_p=0.95, top_k=20. No-think: temperature=0.7, top_p=0.8, presence_penalty=1.5.
| Base | Qwen/Qwen3.8-27B |
| Bits | 4-bit weights (INT4), activations BF16 |
| Algorithm | Intel AutoRound 0.14.2 |
| Group | 128 |
| Packing | compressed-tensors, pack-quantized |
| Kernel | Marlin (Ampere+) |
| Calibration | NeelNanda/pile-10k, 128 samples, seq 2048, 200 iters |
| Left in BF16 | recurrence (in_proj_a / ), vision, MTP, lm_head |
Is this 4-bit?
Yes. INT4 weights. W4A16 means 4-bit weights, 16-bit activations.
Is this AWQ?
No. AutoRound. AWQ is a different 4-bit method — see cyankiwi.
3090 or 5090?
3090 / 4090 / A100 → this. Blackwell → NVFP4 will be faster on those tensor cores.
Apache 2.0, same as the base.
in_proj_b