GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
GotoAI-Inc
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
GotoAI-Inc
Model Tree
Qwen/Qwen3.6-27B
Input Modalities
Output Modalities
Supported Functionality
Weights were quantized from bfloat16 to int4, group size 128, symmetric, weight-only
(activations stay 16-bit) using llmcompressor.model_free_ptq. No calibration data was
used and the model was never loaded — the quantizer operates directly on the safetensors.
Architecture, tokenizer, chat template and processor configs are the vendor's, unmodified.
496 Linear modules were converted, covering 64.7% of the checkpoint's bytes:
| component | precision | size |
|---|---|---|
| language-model linears (64 layers) | int4 g128 | 12.56 GB (64.7%) |
embed_tokens + lm_head (untied) | bfloat16 | 5.09 GB (26.2%) |
vision tower (model.visual, 27 layers) | bfloat16 | 0.92 GB (4.7%) |
MTP speculator head (mtp.*) | bfloat16 | 0.85 GB (4.4%) |
| conv1d kernels, norms, biases | bfloat16 | 0.01 GB |
| total | 19.42 GB |
Four things are deliberately left at 16-bit:
model.visual.* — vLLM builds multimodal towers with quant_config=None, so a
checkpoint carrying quantized vision weights cannot be loaded.mtp.* — the built-in multi-token-prediction speculator head
(mtp_num_hidden_layers: 1), loaded through vLLM's speculative-decoding path rather
than the main stack.linear_attn.conv1d — 3-D causal-convolution kernels in the gated-DeltaNet blocks,
shape (10240, 1, 4). Not Linear layers, and quantizers reject them outright.lm_head + embed_tokens — precision-sensitive, and lm_head is untied here.The gated-DeltaNet projections (in_proj_qkv, in_proj_a, in_proj_b, in_proj_z,
out_proj) are quantized; only the convolution kernels beside them are excluded, along
with the 1-D A_log and dt_bias state-space parameters, which any quantizer skips
automatically.
Runs on released vLLM. Qwen3.6 reuses the Qwen3.5 architecture
(Qwen3_5ForConditionalGeneration, model_type: qwen3_5), which has been supported since
0.25.1 — no nightly build is required:
bash
vllm serve GotoAI-Inc/Qwen3.6-27B-W4A16 \--max-model-len 65536 \--enable-auto-tool-choice --tool-call-parser qwen3_coder \--reasoning-parser qwen3
Do not pass --quantization; compressed-tensors is detected from config.json. The int4
W4A16 scheme uses Marlin kernels and runs on compute capability 7.5 and above.
--tool-call-parser qwen3_coder is what the base model card specifies. In current
vLLM qwen3_xml is an alias for the same parser class, so either name works; without
one, the <tool_call><function=…><parameter=…> XML the chat template asks for is
returned as plain text.--reasoning-parser qwen3 splits <think>…</think> into reasoning_content.--language-model-only skips the vision tower and its multimodal profiling, freeing
~0.92 GB of weights plus the profiling headroom, at the cost of image and video input.--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'. The base card
writes "method": "qwen3_next_mtp"; current vLLM deprecates the per-family names and
normalizes them to mtp, resolving to from the checkpoint's own
config. vLLM aligns the draft's quantization with the target's, and the
entry in this checkpoint's ignore list keeps those tensors bf16. Not smoke-tested here.Only 16 of the 64 layers use full attention (every 4th; the other 48 are gated DeltaNet
with constant-size recurrent state). With 4 KV heads at head_dim 256, the KV cache costs
~64 KB/token — about 2 GB at 32k and 4 GB at 64k. Against 19.42 GB of weights that is
comfortable on 32 GB and above; on a 24 GB card, budget for roughly 32k of context, or add
--language-model-only for more. This is arithmetic from config.json, not a measured
deployment.
Qwen3.6 thinks by default and does not support the /think and /nothink soft
switches. It also has no reasoning_effort knob — the two template variables it does
accept are passed through chat_template_kwargs:
json
{"chat_template_kwargs": {"enable_thinking": false}} // instruct / non-thinking mode{"chat_template_kwargs": {"preserve_thinking": true}} // keep earlier turns' thinking
preserve_thinking is the feature this release adds: by default only the thinking from
the latest user message is retained, and turning it on keeps historical reasoning traces
in context — the base model card recommends it for agentic use, where it improves decision
consistency and KV-cache reuse. Set a server-wide default with
--default-chat-template-kwargs '{"preserve_thinking": true}'; request-level values still
win.
Sampling, per the base model card: temperature=1.0, top_p=0.95, top_k=20 for thinking
mode, temperature=0.6 for precise coding, and
markdown
temperature=0.7, top_p=0.80, presence_penalty=1.5
262144 tokens natively. The base model card documents a YaRN recipe reaching 1,010,000
tokens via --hf-overrides plus VLLM_ALLOW_LONG_MAX_MODEL_LEN=1; rope_type is left at
default here, and static YaRN costs quality at short contexts, so enable it only if you
need it.
Built with llm-quantizer:
bash
./llmq.py run --profile qwen3.6-27b
which is equivalent to:
python
# llmcompressor==0.13.1a20260814, compressed-tensors==0.18.1a20260818,# transformers==5.15.1, torch==2.13.0from llmcompressor import model_free_ptqmodel_free_ptq(model_stub="Qwen/Qwen3.6-27B",save_directory="Qwen3.6-27B-W4A16",scheme="W4A16",ignore=["re:.*visual.*", "re:.*mtp.*", "re:.*\\.conv1d$","lm_head", "re:.*embed_tokens.*"],device="cuda:0",)
The source ships as 15 shards of ~4 GB, and a job holds one shard at a time, so the build peaks at a few GB of VRAM — no re-sharding needed and no large GPU required.
No benchmarks have been run. Data-free round-to-nearest quantization degrades quality more than a calibrated (GPTQ/AWQ) or QAT build; how much, for your task, is unmeasured here. Treat the published Qwen3.6 numbers as describing the bfloat16 model, not this one.
For an agentic model the informative checks are well-formed reasoning_content and clean
multi-step tool calls rather than perplexity: structured emission degrades before fluency
does.
Apache 2.0, inherited from the base model — the vendor's LICENSE is included unmodified.
"Qwen" is Alibaba's mark; this repository is not endorsed by or affiliated with Alibaba.
qwen3_5Qwen3_5MTPre:.*mtp.*