GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
Qwen/Qwen3.5-397B-A17B
Input Modalities
Output Modalities
Supported Functionality
This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert). Compared with keeping the shared expert in bf16, this further reduces the bf16 footprint and improves decode throughput, with no measurable accuracy loss on GSM8K (see Evaluation).
The model was quantized from Qwen/Qwen3.5-397B-A17B using AMD-Quark. Weights and activations are quantized to OCP MXFP4.
Quantization scripts:
markdown
MODEL_PATH="Qwen/Qwen3.5-397B-A17B"OUTPUT_DIR="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2"cd /path/to/Quarkpython3 examples/torch/language_modeling/llm_ptq/quantize_quark.py \--model_dir "${MODEL_PATH}" \--quant_scheme mxfp4 \--layer_quant_scheme '*linear_attn*' ptpc_fp8 \--layer_quant_scheme '*self_attn*' ptpc_fp8 \--no_trust_remote_code \--model_export hf_format \--skip_evaluation \--multi_gpu balanced \--exclude_layers "lm_head" "*mlp.gate" "mtp.*" "model.visual.*" "*shared_expert_gate*" \--output_dir "${OUTPUT_DIR}"
For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.
The model was evaluated on the GSM8K benchmark using the SGLang framework
with lm-evaluation-harness, following the
InferenceX SGLang evaluation recipe
(local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the
original Qwen/Qwen3.5-397B-A17B checkpoint,
evaluated with the identical recipe on SGLang.
| Benchmark | Qwen/Qwen3.5-397B-A17B | amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 (this model) | Recovery |
|---|---|---|---|
| gsm8k (flexible-extract, 5-shot) | 97.71 | 97.24 | 99.55% |
GSM8K is essentially lossless after quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel.
The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe.
markdown
export SGLANG_USE_AITER=1export SGLANG_USE_AITER_UNIFIED_ATTN=1export AITER_FLYDSL_FORCE=1export SGLANG_MAMBA_SSM_DTYPE=bfloat16export SGLANG_USE_AITER_FP8_PER_TOKEN=1python3 -m sglang.launch_server \--model-path amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2 \--trust-remote-code \--host 0.0.0.0 \--port 30000 \--tensor-parallel-size 2 \--attention-backend aiter \--mem-fraction-static 0.8 \--model-loader-extra-config '{"enable_multithread_load": true}' \--watchdog-timeout 1200 \--disable-radix-cache \--reasoning-parser qwen3 \--page-size 16 \--enable-aiter-allreduce-fusion
gsm8k.yaml, whose only change vs. the stock
lm-eval task is a doc_to_text that asks the model to end its response with #### <number> so the
reasoning model's final answer is parsed correctly:markdown
lm_eval --model local-chat-completions --apply_chat_template \--tasks gsm8k.yaml \--num_fewshot 5 \--model_args "model=amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V2,base_url=http://127.0.0.1:30000/v1/chat/completions,num_concurrent=64,tokenized_requests=False,max_length=16384" \--gen_kwargs "max_tokens=12288,temperature=0,top_p=1"
Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.