GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
Qwen/Qwen3.5-397B-A17B
Input Modalities
Output Modalities
Supported Functionality
This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert), and by quantizing the MTP experts to MXFP4 and the MTP self-attention to FP8.
The model was quantized from Qwen/Qwen3.5-397B-A17B using AMD-Quark. Weights and activations are quantized to OCP MXFP4.
Quantization scripts:
text
MODEL_PATH="Qwen/Qwen3.5-397B-A17B"OUTPUT_DIR="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3" cd /path/to/Quark python3 examples/torch/language_modeling/llm_ptq/quantize_quark.py \ --model_dir "${MODEL_PATH}" \ --quant_scheme mxfp4 \ --layer_quant_scheme '*linear_attn*' ptpc_fp8 \ --layer_quant_scheme '*self_attn*' ptpc_fp8 \ --quant_flow file2file \ --no_trust_remote_code \ --model_export hf_format \ --skip_evaluation \ --multi_gpu balanced \ --exclude_layers "lm_head" "*mlp.gate" "mtp.fc" "model.visual.*" "*shared_expert_gate*" "*linear_attn.conv1d*" \ --output_dir "${OUTPUT_DIR}"For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.
The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness, following the InferenceX SGLang evaluation recipe (local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the original Qwen/Qwen3.5-397B-A17B checkpoint, evaluated with the identical recipe on SGLang.
This model can be deployed efficiently using the SGLang backend based on the Docker image lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260922.
| Benchmark | Qwen/Qwen3.5-397B-A17B | amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3 (this model) | Recovery |
|---|---|---|---|
| gsm8k (flexible-extract, 5-shot) | 97.71 | 97.57 | 99.86% |
The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe. Both sides were served with EAGLE MTP.
text
export SGLANG_USE_AITER=1export SGLANG_USE_AITER_UNIFIED_ATTN=1export AITER_FLYDSL_FORCE=1export SGLANG_MAMBA_SSM_DTYPE=bfloat16export SGLANG_USE_AITER_FP8_PER_TOKEN=1 python3 -m sglang.launch_server \ --model-path amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 4 \ --attention-backend aiter \ --mem-fraction-static 0.90 \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --watchdog-timeout 1200 \ --disable-radix-cache \ --reasoning-parser qwen3 \ --page-size 16 \ --enable-aiter-allreduce-fusion \ --enable-metrics \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4doc_to_text that asks the model to end its response with #### <number> so the reasoning model's final answer is parsed correctly:text
lm_eval --model local-chat-completions --apply_chat_template \ --tasks gsm8k.yaml \ --num_fewshot 5 \ --model_args "model=amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=16,tokenized_requests=False,max_length=16384" \ --gen_kwargs "max_tokens=12288,temperature=0,top_p=1"Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.
MODEL_PATH="Qwen/Qwen3.5-397B-A17B"OUTPUT_DIR="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3"cd /path/to/Quarkpython3 examples/torch/language_modeling/llm_ptq/quantize_quark.py \ --model_dir "${MODEL_PATH}" \ --quant_scheme mxfp4 \ --layer_quant_scheme '*linear_attn*' ptpc_fp8 \ --layer_quant_scheme '*self_attn*' ptpc_fp8 \ --quant_flow file2file \ --no_trust_remote_code \ --model_export hf_format \ --skip_evaluation \ --multi_gpu balanced \ --exclude_layers "lm_head" "*mlp.gate" "mtp.fc" "model.visual.*" "*shared_expert_gate*" "*linear_attn.conv1d*" \ --output_dir "${OUTPUT_DIR}"export SGLANG_USE_AITER=1export SGLANG_USE_AITER_UNIFIED_ATTN=1export AITER_FLYDSL_FORCE=1export SGLANG_MAMBA_SSM_DTYPE=bfloat16export SGLANG_USE_AITER_FP8_PER_TOKEN=1python3 -m sglang.launch_server \ --model-path amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3 \ --trust-remote-code \ --host 0.0.0.0 \ --port 8000 \ --tensor-parallel-size 4 \ --attention-backend aiter \ --mem-fraction-static 0.90 \ --model-loader-extra-config '{"enable_multithread_load": true}' \ --watchdog-timeout 1200 \ --disable-radix-cache \ --reasoning-parser qwen3 \ --page-size 16 \ --enable-aiter-allreduce-fusion \ --enable-metrics \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4lm_eval --model local-chat-completions --apply_chat_template \ --tasks gsm8k.yaml \ --num_fewshot 5 \ --model_args "model=amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=16,tokenized_requests=False,max_length=16384" \ --gen_kwargs "max_tokens=12288,temperature=0,top_p=1"