GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
This is a quantized version of Qwen3.6-35B-A3B created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.
The model was quantized from Qwen3.6-35B-A3B using LLM Compressor via the Round-to-Nearest (RTN) algorithm. This reduces the model weights from 67.0 GiB to 35.1 GiB on disk (~48% reduction).
compressed-tensors, num_bits=8, type=int, symmetric=truemlp.gate), shared_expert_gate, linear-attention modules, the visual tower, and lm_head. The routed expert projections are quantized.python
import torchfrom transformers import AutoProcessor, AutoTokenizer, Qwen3_5MoeForConditionalGenerationfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization import QuantizationModifiermodel_id = "Qwen/Qwen3.6-35B-A3B"output_dir = "./Qwen3.6-35B-A3B-w8a8-llmcompressor-v0.12.0"# Step 1: Load the BF16 model and tokenizer.# Load the top-level Qwen3_5MoeForConditionalGeneration rather than AutoModelForCausalLM,# which would demote config.json to the text-only Qwen3_5MoeTextConfig and produce a# checkpoint vLLM rejects.model = Qwen3_5MoeForConditionalGeneration.from_pretrained(model_id,dtype=torch.bfloat16,device_map="cpu",trust_remote_code=True,)tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)# Step 2: Define the W8A8 reciperecipe = QuantizationModifier(scheme="W8A8",targets=["Linear"],ignore=[r"re:.*lm_head",r"re:.*mlp.gate$", # Qwen MoE routerr"re:visual.*",r"re:model.visual.*",r"re:.*embed_tokens$",r"re:.*shared_expert_gate$",r"re:.*linear_attn.*",],)# Step 3: One-shot quantize and save in compressed-tensors formatoneshot(model=model,recipe=recipe,tokenizer=tokenizer,output_dir=output_dir,trust_remote_code_model=True,)# oneshot does not save the processor; multimodal checkpoints need it for vLLM.processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)processor.save_pretrained(output_dir)# Smoke testinputs = tokenizer("What are we having for dinner?", return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0], skip_special_tokens=True))
python
from vllm import LLM, SamplingParamsmodel = LLM(model="amd/Qwen3.6-35B-A3B-w8a8-llmcompressor-v0.12.0",dtype="bfloat16",)sampling_params = SamplingParams(temperature=0.7, max_tokens=256)outputs = model.generate(["Hello, how are you?"], sampling_params)print(outputs[0].outputs[0].text)
markdown
torch==2.11.0zentorch==2.11.0.3vllm==0.26.0llmcompressor==0.12.0
For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):
bash
# Using LLVM OpenMP (llvmopenmp)export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)# Or using Intel OpenMP (libiomp)export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)
Note: Set
LD_PRELOADbefore launching vLLM or any inference script.
The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.
| Benchmark | BF16 Baseline | W8A8 (this model) | Recovery |
|---|---|---|---|
| GSM8K (5-shot) | 0.9629 | 0.9629 | 100% |
Evaluation results will be updated after benchmarking.
bash
lm_eval \--model vllm \--model_args pretrained=amd/Qwen3.6-35B-A3B-w8a8-llmcompressor-v0.12.0,tokenizer=Qwen/Qwen3.6-35B-A3B,dtype=bfloat16,max_model_len=4096,language_model_only=True,enable_thinking=False \--tasks gsm8k \--batch_size auto \--trust_remote_code \--num_fewshot 5 \--apply_chat_template \--log_samples \--gen_kwargs "max_gen_toks=2048" \--output_path .
This model is distributed under the same license as the source model. See the LICENSE file for details.
Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.