GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
inference-optimization
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
inference-optimization
Model Tree
Qwen/Qwen3.5-397B-A17B
Input Modalities
Output Modalities
Supported Functionality
This model is a quantized version of Qwen/Qwen3.5-397B-A17B. It was evaluated on several tasks to assess its quality in comparison to the unquantized model.
This model was obtained by quantizing the weights and activations of Qwen/Qwen3.5-397B-A17B to FP8 data type, ready for inference with vLLM.
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
Only the weights and activations of the linear operators within transformer blocks are quantized using LLM Compressor. Layers such as the visual encoder, linear attention (Gated DeltaNet), MoE router gates, shared experts, and token embeddings are kept in original precision.
This model can be deployed efficiently using vLLM.
markdown
vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --language-model-only --tensor-parallel-size 4
markdown
vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --tensor-parallel-size 4
markdown
vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder --tensor-parallel-size 4
markdown
vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' --tensor-parallel-size 4
Send requests to the server:
python
from openai import OpenAIopenai_api_key = "EMPTY"openai_api_base = "http://<your-server-host>:8000/v1"client = OpenAI(api_key=openai_api_key,base_url=openai_api_base,)model = "RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic"messages = [{"role": "user", "content": "Explain quantum mechanics clearly and concisely."},]outputs = client.chat.completions.create(model=model,messages=messages,)generated_text = outputs.choices[0].message.contentprint(generated_text)
This model was created by applying LLM Compressor with FP8 dynamic quantization, as presented in the code snippet below.
python
from transformers import AutoProcessor, Qwen3_5MoeForConditionalGenerationfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifierMODEL_ID = "Qwen/Qwen3.5-397B-A17B"# Load model.model = Qwen3_5MoeForConditionalGeneration.from_pretrained(MODEL_ID, dtype="auto")processor = AutoProcessor.from_pretrained(MODEL_ID)# Configure the quantization algorithm and scheme.# In this case, we:# * quantize the weights to fp8 with channel-wise quantization# * quantize the activations to fp8 with dynamic per-token quantizationrecipe = QuantizationModifier(targets="Linear",scheme="FP8_DYNAMIC",ignore=["re:.*lm_head","re:visual.*","re:model.visual.*","re:.*mlp.gate$","re:.*embed_tokens$","re:.*shared_expert_gate$","re:.*mlp\\.shared_expert$","re:.*linear_attn.*",],)# Apply quantization.oneshot(model=model, recipe=recipe)# Save to disk in compressed-tensors format.SAVE_DIR = MODEL_ID.rstrip("/").split("/")[-1] + "-FP8-dynamic"model.save_pretrained(SAVE_DIR)processor.save_pretrained(SAVE_DIR)
This model was evaluated on GSM8K-Platinum, MMLU-Pro, IFEval, Math 500, GPQA Diamond, AIME 25, and LiveCodeBench v6 using lm-evaluation-harness and lighteval, served with vLLM using --language-model-only.
The results were obtained using the following commands:
The model was served with vLLM using the following command:
markdown
vllm serve RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic --reasoning-parser qwen3 --language-model-only --max-model-len 96000 --tensor-parallel-size 4
Each benchmark was run 3 times with different seeds (42, 1234, 4158), except AIME 25 which used 8 seeds (42, 1234, 4158, 5322, 1356, 9843, 3344, 5678). Scores are averaged across all seeds.
markdown
lm_eval --model local-chat-completions \--tasks gsm8k_platinum_cot_llama \--model_args "model=RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \--num_fewshot 0 \--apply_chat_template \--output_path results.json \--seed 42 \--gen_kwargs "do_sample=true,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=0.0,repetition_penalty=1.0,seed=42"
markdown
lm_eval --model local-chat-completions \--tasks ifeval \--model_args "model=RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=2400" \--apply_chat_template \--output_path results.json \--seed 42 \--gen_kwargs "do_sample=true,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=0.0,repetition_penalty=1.0,seed=42"
markdown
lm_eval --model local-chat-completions \--tasks mmlu_pro_chat \--model_args "model=RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic,max_length=96000,base_url=http://0.0.0.0:8000/v1/chat/completions,num_concurrent=128,max_retries=3,tokenized_requests=False,tokenizer_backend=None,timeout=3600" \--num_fewshot 0 \--apply_chat_template \--output_path results.json \--seed 42 \--gen_kwargs "do_sample=true,temperature=0.6,top_p=0.95,top_k=20,min_p=0.0,max_gen_toks=64000,presence_penalty=0.0,repetition_penalty=1.0,seed=42"
litellm_config.yaml:
yaml
model_parameters:provider: "hosted_vllm"model_name: "hosted_vllm/RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic"base_url: "http://0.0.0.0:8000/v1"api_key: ""timeout: 3600concurrent_requests: 32generation_parameters:temperature: 0.6max_new_tokens: 64000top_p: 0.95top_k: 20min_p: 0.0presence_penalty: 0.0repetition_penalty: 1.0seed: 0
markdown
lighteval endpoint litellm litellm_config.yaml \"math_500|0,gpqa:diamond|0,lcb:codegeneration_v6|0" \--output-dir results \--save-details
markdown
lighteval endpoint litellm litellm_config.yaml \"aime25|0" \--output-dir results \--save-details