Qwen3.5-397B-A17B-MoE-MXFP4

Model Overview

• Model Architecture: Qwen3_5MoeForConditionalGeneration
  • Input: Text, Image, Video
  • Output: Text
• Supported Hardware Microarchitecture: AMD MI350 / MI355
• ROCm: 7.2.0
• PyTorch: 2.9.1
• Transformers: 5.3.0
• Operating System(s): Linux
• Inference Engine: SGLang
• Model Optimizer: AMD-Quark (v0.12)
  • Quantized layers: All MoE experts in the language model, including the shared expert (the shared expert is also fused into the MoE kernel for faster decode).
  • Weight quantization: OCP MXFP4, Static
  • Activation quantization: OCP MXFP4, Dynamic

This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert). Compared with keeping the shared expert in bf16, this further reduces the bf16 footprint and improves decode throughput, with no measurable accuracy loss on GSM8K (see Evaluation).

Model Quantization

The model was quantized from Qwen/Qwen3.5-397B-A17B-FP8 using AMD-Quark. Weights and activations are quantized to OCP MXFP4.

Quantization scripts:

markdown
Copy code
import os
from quark.torch import LLMTemplate, ModelQuantizer

# Register qwen3_5_moe template
qwen3_5_moe_template = LLMTemplate(
    model_type="qwen3_5_moe",
    kv_layers_name=["*k_proj", "*v_proj"],
    q_layer_name="*q_proj"
)
LLMTemplate.register_template(qwen3_5_moe_template)

# Configuration
ckpt_path = "Qwen/Qwen3.5-397B-A17B-FP8"
output_dir = "amd/Qwen3.5-397B-A17B-MoE-MXFP4"
quant_scheme = "mxfp4"
# NOTE: shared expert is NOT excluded here, so it is quantized to MXFP4 as well.
exclude_layers = ["lm_head", "model.visual.*", "mtp.*", "*mlp.gate", "*shared_expert_gate*", "*.linear_attn.*", "*.self_attn.*"]

# Get quant config from template
template = LLMTemplate.get("qwen3_5_moe")
quant_config = template.get_config(scheme=quant_scheme, exclude_layers=exclude_layers)

# Quantize with file-to-file mode
quantizer = ModelQuantizer(quant_config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=ckpt_path,
    save_path=output_dir,
)

For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.

Evaluation

The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness, following the InferenceX SGLang evaluation recipe (local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the original Qwen/Qwen3.5-397B-A17B-FP8 checkpoint, evaluated with the identical recipe on SGLang.

Accuracy

GSM8K is essentially lossless after quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel.

Reproduction

The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe.

1. Serve the model with SGLang:

markdown
Copy code
python3 -m sglang.launch_server \
    --model-path amd/Qwen3.5-397B-A17B-MoE-MXFP4 \
    --tensor-parallel-size 4 \
    --trust-remote-code \
    --attention-backend aiter \
    --mem-fraction-static 0.8 \
    --host 0.0.0.0 --port 30000

2. Run lm-evaluation-harness against the running server (chat-completions endpoint, chat template applied, 5-shot, greedy). This uses InferenceX's gsm8k.yaml, whose only change vs. the stock lm-eval task is a doc_to_text that asks the model to end its response with #### <number> so the reasoning model's final answer is parsed correctly:

markdown
Copy code
lm_eval --model local-chat-completions --apply_chat_template \
  --tasks gsm8k.yaml \
  --num_fewshot 5 \
  --model_args "model=amd/Qwen3.5-397B-A17B-MoE-MXFP4,base_url=http://127.0.0.1:30000/v1/chat/completions,num_concurrent=64,tokenized_requests=False,max_length=16384" \
  --gen_kwargs "max_tokens=12288,temperature=0,top_p=1"

License

Apache-2.0. Modifications Copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.