GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
amd-satre
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd-satre
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
Quark's built-in qwen3_5_moe recipe quantizes the routed MoE experts only — 30,720
linears (40 layers x 256 experts x 3 projections). 462 modules are excluded and remain
BF16:
lm_head, model.visual.* (vision tower), mtp.*, *mlp.gate, *shared_expert_gate*, *.linear_attn.*, *.self_attn.*, *.shared_expert.*
Because attention, the shared experts, the router and the vision tower all stay BF16, the size reduction is smaller than a whole-model quantization would give.
bash
python examples/torch/language_modeling/llm_ptq/quantize_quark.py \--model_dir Qwen/Qwen3.6-35B-A3B \--device cuda --data_type bfloat16 --model_attn_implementation eager \--dataset pileval --num_calib_data 128 --seq_len 512 --batch_size 1 \--quant_scheme mxfp4 \--quant_algo awq --quant_algo_config_file awq <config.json> \--model_export hf_format --export_weight_format real_quantized \--output_dir <out> --skip_evaluation
0.13+50dac6e159a.pileval (mit-han-lab/pile-val-backup), 128 samples, seq_len=512.MODEL_ARCH.LLAMA and supports asymmetric INT4 per-group-32 weight-only on Llama2/Llama3, while this model is Qwen3_5MoeForConditionalGeneration. Safetensors is published instead.GSM8K (5-shot, max_gen_toks=2048) and WikiText-2 perplexity, measured with lm-evaluation-harness against the bf16 baseline (Qwen/Qwen3.6-35B-A3B).
| Metric | bf16 baseline | this checkpoint | recovery |
|---|---|---|---|
| GSM8K flexible-extract | 0.3798 | 0.5997 | 157.9% |
| GSM8K strict-match | 0.3821 | 0.6240 | 163.3% |
| WikiText-2 word_perplexity | 8.1511 | 8.4724 | 96.2% |
Caveat on baseline methodology: the bf16 baselines above were originally distorted by two independent bugs we found and fixed before finalizing these numbers: (1) lm-evaluation-harness's default max_gen_toks=256 truncated this thinking model's <think> traces before reaching an answer (fixed: raised to 2048); (2) a real, reproducible bug in vLLM's Triton implementation of the chunk_gated_delta_rule custom op on this ROCm build causes some bf16 generations to collapse into a fixed degenerate phrase and stop early -- confirmed via a working reference PyTorch implementation (transformers eager, no custom kernel) producing coherent output on the identical prompt/weights, and confirmed in both eager and CUDA-graph-compiled vLLM modes (ruling out graph capture as the cause). Workaround: compilation_config={"custom_ops": ["-chunk_gated_delta_rule"]} forces the correct fallback path. Cross-validated against a 150-example CPU ground-truth run for both base models -- the bug's net effect on the aggregate GSM8K score was small, so the baseline numbers above (measured with the workaround applied) should be reliable. Separately, several checkpoints in this family genuinely score well above the bf16 baseline on GSM8K (a 15-45 point jump in some cases). We verified this is real, not a measurement artifact: it persists identically with and without the kernel-bug workaround applied. It most likely reflects quantization noise suppressing this reasoning model's tendency to enter long, sometimes unproductive chains of self-questioning on the unquantized baseline -- not a genuine capability improvement from quantization. WikiText-2 perplexity (likelihood-based, not generation-based, and unaffected by either bug) shows the expected small regression from quantization and is the more trustworthy signal of quantization quality here.