GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Model Details
Model Provider
amd
Model Tree
Qwen/Qwen3.6-35B-A3B
Input Modalities
Output Modalities
Supported Functionality
This is a quantized version of Qwen3.6-35B-A3B created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.
The model was quantized from Qwen3.6-35B-A3B using LLM Compressor via the GPTQ algorithm. This reduces the model weights from 67.0 GiB to 18.1 GiB on disk (~73% reduction).
compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128group_size=128, actorder=static), stored as pack-quantizedmlp.gate), lm_head, embed_tokens, and the layer norms. The 256 routed experts, the shared expert, and the linear-attention projections are all quantized, which is why the reduction is close to the 75% ideal for INT4.python
import torchfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom datasets import load_datasetfrom llmcompressor import oneshotfrom llmcompressor.modifiers.quantization.gptq import GPTQModifiermodel_id = "Qwen/Qwen3.6-35B-A3B"save_dir = "./Qwen3.6-35B-A3B-w4a16-llmcompressor-v0.12.0"CALIB_SIZE = 128MAX_SEQ_LENGTH = 2048# Step 1: Load the BF16 model and tokenizermodel = AutoModelForCausalLM.from_pretrained(model_id,device_map="cpu",dtype=torch.bfloat16,trust_remote_code=True,)tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations to# build the per-layer Hessians used to compensate the rounding error.ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{CALIB_SIZE}]")ds = ds.map(lambda example: {"text": "\n".join(m["content"] for m in example["messages"] if m["content"])},remove_columns=ds.column_names,)# Step 3: Define the W4A16 GPTQ recipe. Routing layers are skipped: they are tiny,# and a mis-routed token costs far more accuracy than 4-bit expert weights do.# The W4A16 preset implies group_size=128, which is valid here because every# quantized weight's column count (2048 hidden, 512 MoE intermediate) divides by 128.recipe = GPTQModifier(targets="Linear",scheme="W4A16",ignore=["lm_head",r"re:.*\.router$",r"re:.*\.router\..*",r"re:.*\.gate$",r"re:.*\.mlp\.gate$",],)# Step 4: One-shot quantize with calibration and save in compressed-tensors formatoneshot(model=model,dataset=ds,recipe=recipe,max_seq_length=MAX_SEQ_LENGTH,processor=tokenizer,)model.save_pretrained(save_dir, save_compressed=True)tokenizer.save_pretrained(save_dir)# Smoke testinputs = tokenizer("What are we having for dinner?", return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0], skip_special_tokens=True))
Run end to end with the driver script:
bash
numactl --physcpubind=0-95 python llm_compressor_quantize_and_run.py \--model_id Qwen/Qwen3.6-35B-A3B \--save_dir ./Qwen3.6-35B-A3B-w4a16-llmcompressor-v0.12.0 \--recipe gptq \--scheme W4A16 \--run
python
from vllm import LLM, SamplingParamsmodel = LLM(model="amd/Qwen3.6-35B-A3B-w4a16-llmcompressor-v0.12.0",dtype="bfloat16",)sampling_params = SamplingParams(temperature=0.7, max_tokens=256)outputs = model.generate(["Hello, how are you?"], sampling_params)print(outputs[0].outputs[0].text)
markdown
torch==2.11.0zentorch==2.11.0.3vllm==0.26.0llmcompressor==0.12.0
For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):
bash
# Using LLVM OpenMP (llvmopenmp)export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)# Or using Intel OpenMP (libiomp)export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)
Note: Set
LD_PRELOADbefore launching vLLM or any inference script.
The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.
| Benchmark | BF16 Baseline | W4A16 (this model) | Recovery |
|---|---|---|---|
| GSM8K (5-shot) | 0.9651 | 0.9636 | 99.84% |
Evaluation results will be updated after benchmarking.
bash
lm_eval \--model vllm \--model_args pretrained=amd/Qwen3.6-35B-A3B-w4a16-llmcompressor,tokenizer=Qwen/Qwen3.6-35B-A3B,dtype=bfloat16,max_model_len=4096,enable_thinking=False \--tasks gsm8k \--batch_size auto \--trust_remote_code \--num_fewshot 5 \--apply_chat_template \--log_samples \--gen_kwargs "max_gen_toks=2048" \--output_path .
AutoModelForCausalLM during quantization, so the saved config is the text-only Qwen3_5MoeForCausalLM variant. The vision path of the source model is not carried over.enable_thinking=False (and a generous max_gen_toks) so answer extraction stays comparable to the BF16 baseline.This model is distributed under the same license as the source model. See the LICENSE file for details.
Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.