GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
yacht
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
yacht
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
Run Qwen3.6-35B-A3B on a single 24 GB GPU (RTX 4090) without losing Thai quality.
Quantized MoE models tend to degrade low-resource languages before English shows any damage. Thai is an abugida — vowels and tone marks are combining codepoints — so damage appears as detached, doubled, or misordered marks while English still looks perfect. This build was calibrated and tested against that specific failure mode.
GPTQ Hessian error-compensation genuinely reshapes weights at 4 bits, so calibration content matters here (unlike FP8, which is data-free). 497 documents / 1,131,013 tokens, 964,929 Thai (85.3%), all from sources permitting commercial use:
| source | licence | share |
|---|---|---|
| Wikipedia (th) | CC-BY-SA-3.0 | 45% |
| C4 (th) | ODC-BY | 35% |
| Thai stress shard — combining marks, tone marks, thanthakhat, sara am, Thai numerals ๐–๙ | authored for this build | 5% |
| Wikipedia (en) — control | CC-BY-SA-3.0 | 15% |
moe_calibrate_all_experts=True, so all 256 experts see every calibration token — no expert is
left with degenerate statistics. dampening_frac=0.01, max_seq_length=2048.
Attribution: C4 — Raffel et al., JMLR 2020, ODC-BY. Wikipedia — © Wikipedia contributors, CC-BY-SA 3.0.
Structural metrics over the Thai Unicode block (U+0E00–U+0E7F): orphan combining marks, marks opening a word, doubled tone marks, tone-before-vowel ordering, orphan leading vowels (เ แ โ ใ ไ), U+FFFD, lone surrogates. Scored against the BF16 original on identical prompts, with an English control so any damage must be shown Thai-specific.
| suite | BF16 | W4A16 |
|---|---|---|
| thai_general | 0 | 0 |
| thai_stress | 1 | 0 |
| thai_domain | 0 | 0 |
| english_control | 0 | 0 |
Long-context test — 8,685-token retrieval-style Thai prompt, the regime where corruption actually shows up:
| arm | corrupted |
|---|---|
| greedy | 0 / 24 |
| sampled (t=0.7, top_p=0.8) | 0 / 24 |
| short-context control | 0 / 24 |
At this model's own generation_config defaults (temperature 1.0, top_k 20, top_p 0.95), Thai
responses contain a duplicated combining mark in roughly 3% of generations — inherited from the
base model, not introduced by quantization. 90 generations per model, identical prompts:
| model | corrupted |
|---|---|
| BF16 (unquantized) | 3/90 — 3.3% |
| FP8 | 3/90 — 3.3% |
| W4A16 (this model) | 4/90 — 4.4% |
At n=90 these are statistically indistinguishable. At temperature 0.7, top_p 0.8 all three score
0/24. If you need structurally reliable Thai, lower the temperature or mask repeated combining marks
with a logits processor — a different quantization will not help.
A combining mark with no Thai on either side counts as a citation, not a defect — when explaining
Thai orthography the model correctly writes marks in isolation, e.g. เครื่องหมาย (์).
bash
MODEL=<ORG>/Qwen3.6-35B-A3B-W4A16 # or a local pathvllm serve "$MODEL" --quantization compressed-tensors \--max-model-len 32768 --max-num-batched-tokens 32768
[!IMPORTANT] Keep
--max-num-batched-tokensabove your longest prompt. 30 of this model's 40 layers are GatedDeltaNet linear attention carrying a recurrent state across the sequence rather than a KV cache. When a prompt exceeds the batched-token budget, chunked prefill splits it and that state must be carried across the boundary. A prompt of 8,537 tokens against--max-num-batched-tokens=8192has been observed to produce duplicated Thai tone marks (ฟื้้ณภาพforฟื้นฟูสมรรถภาพ). Short prompts never show it. Raising the budget above the longest prompt avoids the split.
Also: do not set --kv-cache-dtype fp8 — it corrupts hybrid GatedDeltaNet models
(vllm#37554). Default BF16 KV cache is correct.
W4A16 MoE routes through Marlin, which needs Ada (SM89) or Hopper (SM90). It crashes on A100 (SM80) — use an INT8 W8A8 build there.
W4A16 group-128 via llmcompressor 0.11.0 GPTQModifier, compressed-tensors /
pack-quantized format. 377 minutes on one H100.
Left in BF16: lm_head, embed_tokens, norms, mlp.gate (the MoE router — quantizing it perturbs
expert routing across every downstream layer), shared_expert_gate, all 30 GatedDeltaNet
linear_attn.* / conv1d layers, the vision tower, and the MTP head. The 10 full-attention layers
(3, 7, 11, 15, 19, 23, 27, 31, 35, 39) have q/k/v/o_proj quantized normally.
Reproducing: oneshot on this multimodal MoE checkpoint mangles keys
(llm-compressor#2568, still present in
0.11.0) — the text prefix repeats and the vision tower nests inside the text model, so vLLM raises
KeyError: '…experts.w2_weight'. This checkpoint has been repaired: a pure rename, tensor data
untouched, key count preserved 1:1. Experts are stored per-expert 2-D rather than fused 3-D — that
is correct, vLLM fuses them at load.