Calibration
128 shuffled Flickr30k
test-split image/first-caption pairs, seed 42, with every expert receiving
calibration inputs. Each example uses up to 256 image tokens and 1024 total
tokens. Dataset revision: 765d117f3eec816f2bfdc2d73ebb50a6f77b86a4.
Each quantization method starts independently from the original BF16 weights.
The recipe and package versions are in recipe.yaml and quantization_run.json.
The source model's fused 3D expert parameters were converted to separate 2D
linear modules. Use AutoModelForImageTextToText with trust_remote_code=True
to load the bundled modeling_qwen3_vl_moe_quantized.py implementation.
Loading directly with the stock Qwen3-VL MoE class does not reproduce this layout.
The validated environment used Torch 2.7.0, Transformers 4.57.1,
compressed-tensors 0.13.0, and Accelerate. Transformers can decompress weights
to BF16 while loading; allow memory for the full BF16 model plus working space.
Saved checkpoint size is not a measurement of runtime GPU memory.
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "CompressedMichael/Qwen3-VL-30B-A3B-Instruct-SmoothQuant-W8A8"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
model_id,
trust_remote_code=True,
dtype=torch.bfloat16,
device_map="auto",
attn_implementation="sdpa",
).eval()
This is a compressed-tensors export. Compatibility with AutoGPTQ serialization,
vLLM serving, and accelerated low-bit kernels has not been validated.
Validation
On September 13, 2026, the local export passed a fresh Transformers reload
with no missing, unexpected, or mismatched weights and no loader errors.
Greedy generation of 48 tokens for the next held-out Flickr30k image produced
a nonempty image description with no NaN generation scores. See validation.json.
artifact_audit.json records quantization coverage, storage dtypes, shard index,
and file completeness checks for all 18,624 quantized linears.
These checks establish reload and generation functionality. Benchmark accuracy,
accuracy relative to the original model, and low-bit serving speed were not
measured in this validation. The calibration and validation images are not
redistributed in this model repository.
License and attribution
The original model is from the Qwen team and is released under Apache-2.0.
This repository provides a quantized derivative under the same license.
See the original model card, LICENSE, and NOTICE.