GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
yacht
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
yacht
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
Run Qwen3.6-35B-A3B on 1× H100 80GB, or 2× RTX 4090 24GB, without losing Thai quality.
None — this build is data-free. FP8 derives weight scales from weight max and computes activation scales per-token at runtime, so no calibration corpus is involved and none influenced these weights. Licence is therefore Apache-2.0, inherited from the base model, with no third-party data encumbrance.
Scored against the BF16 original on identical prompts, using structural metrics over the Thai Unicode block (U+0E00–U+0E7F): orphan combining marks, marks opening a word, doubled tone marks, tone-before-vowel ordering, orphan leading vowels (เ แ โ ใ ไ), U+FFFD, lone surrogates.
Structural defects, lower is better:
| suite | BF16 | FP8 |
|---|---|---|
| thai_general | 0 | 0 |
| thai_stress | 1 | 0 |
| thai_domain | 0 | 0 |
| english_control | 0 | 0 |
No Thai degradation relative to the BF16 original. Manually confirmed: correct thanthakhat (สิทธิ์ ศักดิ์ องค์), full Thai numeral range ๐–๙ (byte-fallback in this tokenizer, the most fragile decode path), correct tone marks and combining vowels.
A combining mark with no Thai character on either side is counted as a citation, not a defect —
when asked to explain Thai orthography the model correctly writes marks in isolation, e.g.
เครื่องหมาย (์). Without that exemption the checker punishes correct pedagogical text and, worse,
scores a healthy model as more broken simply because it explained more characters. BF16 emitted 3
such citations here, FP8 one.
Long-context test — 8,685-token retrieval-style Thai prompt: 0/24 corrupted greedy, 0/24 sampled (t=0.7, top_p=0.8), 0/24 short-context control. BF16 scores identically.
At this model's own generation_config defaults (temperature 1.0, top_k 20, top_p 0.95), Thai
responses contain a duplicated combining mark in roughly 3% of generations — and this is inherited
from the base model, not introduced by quantization. 90 generations per model, identical prompts:
| model | corrupted |
|---|---|
| BF16 (unquantized) | 3/90 — 3.3% |
| FP8 | 3/90 — 3.3% |
| W4A16 | 4/90 — 4.4% |
At temperature 0.7, top_p 0.8 all three score 0/24. If you need Thai output to be structurally
reliable, lower the temperature or mask repeated combining marks with a logits processor — do not
expect a different quantization to help.
bash
MODEL=<ORG>/Qwen3.6-35B-A3B-FP8 # or a local path to this checkpoint# 1× H100 80GBvllm serve "$MODEL" --quantization compressed-tensors --max-model-len 40960# 2× RTX 4090 24GBvllm serve "$MODEL" --quantization compressed-tensors \--tensor-parallel-size 2 --max-model-len 16384
python
from vllm import LLMllm = LLM(model="<ORG>/Qwen3.6-35B-A3B-FP8", quantization="compressed-tensors", dtype="bfloat16")
Needs Ada (SM89) or Hopper (SM90). On Ampere (A100) there are no FP8 tensor cores — vLLM
dequantizes to BF16, giving quantization error with no speedup; use INT8 W8A8 there.
Do not set --kv-cache-dtype fp8 — it corrupts hybrid GatedDeltaNet models
(vllm#37554).
FP8_BLOCK scheme via llmcompressor 0.11.0 — E4M3, block-[128,128] weights, dynamic per-token
activations, compressed-tensors / float-quantized format.
Left in BF16: lm_head, embed_tokens, norms, mlp.gate (the MoE router — quantizing it perturbs
expert routing across every downstream layer), shared_expert_gate, all 30 GatedDeltaNet
linear_attn.* / conv1d layers, the vision tower (333 tensors), and the MTP head. The 10
full-attention layers (3, 7, 11, 15, 19, 23, 27, 31, 35, 39) have q/k/v/o_proj quantized normally.
Reproducing: oneshot on this multimodal MoE checkpoint mangles keys
(llm-compressor#2568, still present in
0.11.0) — the text prefix repeats (model.language_model.language_model.language_model.…) and the
vision tower nests inside the text model, so vLLM raises KeyError: '…experts.w2_weight'. This
checkpoint has been repaired: a pure rename, tensor data untouched, key count preserved 1:1.
Experts are stored per-expert 2-D (61,440 tensors) rather than fused 3-D — that is correct, vLLM
fuses them at load.