GLM-5.2 is live. #1 throughput on OpenRouter, pay-per-token on FriendliAI. Try it today ➜
amd
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
amd
Model Tree
Qwen/Qwen3-VL-8B-Instruct
Input Modalities
Output Modalities
Supported Functionality
This is a quantized version of Qwen3-VL-8B-Instruct created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.
The model was quantized from Qwen3-VL-8B-Instruct using LLM Compressor via the GPTQ algorithm. This reduces the model weights from 16.3 GiB to 6.7 GiB on disk (~59% reduction).
compressed-tensors, num_bits=4, type=int, symmetric=true, group_size=128group_size=128, actorder=static), stored as pack-quantizedmodel.visual.*) and lm_head. Only the language-model Linear layers are quantized, so a text-only calibration set exercises exactly the modules being quantized.python
import torchfrom transformers import AutoProcessor, AutoTokenizer, Qwen3VLForConditionalGenerationfrom datasets import load_datasetfrom llmcompressor import oneshotfrom llmcompressor.modifiers.gptq import GPTQModifiermodel_id = "Qwen/Qwen3-VL-8B-Instruct"output_dir = "./Qwen3-VL-8B-Instruct-w4a16-llmcompressor-v0.12.0"NUM_CALIBRATION_SAMPLES = 128MAX_SEQUENCE_LENGTH = 2048# Step 1: Load the BF16 model and tokenizer.# Load the top-level Qwen3VLForConditionalGeneration rather than AutoModelForCausalLM,# which would demote config.json to the inner text-only LM and produce a checkpoint# vLLM rejects.model = Qwen3VLForConditionalGeneration.from_pretrained(model_id,dtype=torch.bfloat16,device_map="cpu",trust_remote_code=True,)tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)# Step 2: Load calibration data. GPTQ is data-driven: it needs real activations to# build the per-layer Hessians used to compensate the rounding error.ds = load_dataset("HuggingFaceH4/ultrachat_200k",split=f"train_sft[:{NUM_CALIBRATION_SAMPLES}]",)ds = ds.map(lambda example: {"text": "\n".join(m["content"] for m in example["messages"] if m["content"])},remove_columns=ds.column_names,)# Step 3: Define the W4A16 GPTQ recipe. The vision tower lives under model.visual.*# and is kept in BF16 along with lm_head.recipe = GPTQModifier(scheme="W4A16",targets="Linear",ignore=[r"re:.*lm_head", r"re:.*visual.*"],)# Step 4: One-shot quantize with calibration and save in compressed-tensors formatoneshot(model=model,dataset=ds,recipe=recipe,max_seq_length=MAX_SEQUENCE_LENGTH,tokenizer=tokenizer,output_dir=output_dir,trust_remote_code_model=True,)# oneshot does not save the processor; multimodal checkpoints need it for vLLM.processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)processor.save_pretrained(output_dir)# Smoke test (text-only)inputs = tokenizer("What are we having for dinner?", return_tensors="pt")output = model.generate(**inputs, max_new_tokens=30)print(tokenizer.decode(output[0], skip_special_tokens=True))
python
from vllm import LLM, SamplingParamsmodel = LLM(model="amd/Qwen3-VL-8B-Instruct-w4a16-llmcompressor-v0.12.0",dtype="bfloat16",)sampling_params = SamplingParams(temperature=0.7, max_tokens=256)outputs = model.generate(["Hello, how are you?"], sampling_params)print(outputs[0].outputs[0].text)
markdown
torch==2.11.0zentorch==2.11.0.3vllm==0.26.0llmcompressor==0.12.0
For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):
bash
# Using LLVM OpenMP (llvmopenmp)export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)# Or using Intel OpenMP (libiomp)export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)
Note: Set
LD_PRELOADbefore launching vLLM or any inference script.
The model was evaluated against the BF16 (unquantized) baseline on multimodal benchmarks using lm-evaluation-harness with the vLLM vision-language engine.
| Benchmark | BF16 Baseline | W4A16 (this model) | Recovery |
|---|---|---|---|
| ChartQA | 0.5544 | 0.5884 | 106.13% |
| MMMU (val, tech & engineering) | 0.3952 | 0.3476 | 87.96% |
bash
lm_eval \--model vllm-vlm \--model_args pretrained=amd/Qwen3-VL-8B-Instruct-w4a16-llmcompressor-v0.12.0,dtype=bfloat16 \--tasks chartqa,mmmu_val_tech_and_engineering \--batch_size auto \--trust_remote_code \--apply_chat_template \--log_samples \--output_path .
This model is distributed under the same license as the source model. See the LICENSE file for details.
Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.