GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
Base
Qwen/Qwen3.5-397B-A17B-FP8
Quantized
this model
Input Modalities
Text
Image
Video
Output Modalities
Text
Supported Functionality
Dedicated Endpoints
This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert). Compared with keeping the shared expert in bf16, this further reduces the bf16 footprint and improves decode throughput, with no measurable accuracy loss on GSM8K (see Evaluation).
The model was quantized from Qwen/Qwen3.5-397B-A17B-FP8 using AMD-Quark. Weights and activations are quantized to OCP MXFP4.
Quantization scripts:
markdown
import osfrom quark.torch import LLMTemplate, ModelQuantizer# Register qwen3_5_moe templateqwen3_5_moe_template = LLMTemplate(model_type="qwen3_5_moe",kv_layers_name=["*k_proj", "*v_proj"],q_layer_name="*q_proj")LLMTemplate.register_template(qwen3_5_moe_template)# Configurationckpt_path = "Qwen/Qwen3.5-397B-A17B-FP8"output_dir = "amd/Qwen3.5-397B-A17B-MoE-MXFP4"quant_scheme = "mxfp4"# NOTE: shared expert is NOT excluded here, so it is quantized to MXFP4 as well.exclude_layers = ["lm_head", "model.visual.*", "mtp.*", "*mlp.gate", "*shared_expert_gate*", "*.linear_attn.*", "*.self_attn.*"]# Get quant config from templatetemplate = LLMTemplate.get("qwen3_5_moe")quant_config = template.get_config(scheme=quant_scheme, exclude_layers=exclude_layers)# Quantize with file-to-file modequantizer = ModelQuantizer(quant_config)quantizer.direct_quantize_checkpoint(pretrained_model_path=ckpt_path,save_path=output_dir,)
For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.
The model was evaluated on the GSM8K benchmark using the SGLang framework
with lm-evaluation-harness, following the
InferenceX SGLang evaluation recipe
(local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the
original Qwen/Qwen3.5-397B-A17B-FP8 checkpoint,
evaluated with the identical recipe on SGLang.
| Benchmark | Qwen/Qwen3.5-397B-A17B-FP8 | amd/Qwen3.5-397B-A17B-MoE-MXFP4 (this model) | Recovery |
|---|---|---|---|
| gsm8k (flexible-extract, 5-shot) | 97.95 | 97.27 | 99.31% |
GSM8K is essentially lossless after quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel.
The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe.
markdown
python3 -m sglang.launch_server \--model-path amd/Qwen3.5-397B-A17B-MoE-MXFP4 \--tensor-parallel-size 4 \--trust-remote-code \--attention-backend aiter \--mem-fraction-static 0.8 \--host 0.0.0.0 --port 30000
gsm8k.yaml, whose only change vs. the stock
lm-eval task is a doc_to_text that asks the model to end its response with #### <number> so the
reasoning model's final answer is parsed correctly:markdown
lm_eval --model local-chat-completions --apply_chat_template \--tasks gsm8k.yaml \--num_fewshot 5 \--model_args "model=amd/Qwen3.5-397B-A17B-MoE-MXFP4,base_url=http://127.0.0.1:30000/v1/chat/completions,num_concurrent=64,tokenized_requests=False,max_length=16384" \--gen_kwargs "max_tokens=12288,temperature=0,top_p=1"
Apache-2.0. Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.