What is quantized
Table with columns: Part, Precision, Why| Part | Precision | Why |
|---|
Routed experts gate_proj / up_proj / down_proj (36,864 modules: 48 layers × 256) | int4, group 128, symmetric | almost all of the bytes; fused MoE WNA16 kernel |
| Attention, Gated DeltaNet (linear attention) layers, shared experts, routers, norms | bf16 | small, and sensitive to int4 |
Embeddings, lm_head, MTP head, vision tower | bf16 | standard |
78.1 GB in all (the bf16 source is about 244 GB). Group 128 divides every quantized input dimension at tensor-parallel size 1 and 2.
How it was made
- Tool: llm-compressor 0.14.0
model_free_ptq with compressed-tensors 0.19.0, data-free (no calibration set), on CPU only (38 cores, 70 min).
- Scheme:
num_bits=4, symmetric=true, strategy=group, group_size=128, observer memoryless-mse; no activation quantization.
- Targets (both the checkpoint's and vLLM's module names):
re:^(?:model\.language_model|language_model\.model)\.layers\.\d+\.mlp\.experts\.\d+\.(?:gate_proj|up_proj|down_proj)$
- Source revision:
dc4d348443bc740c68e2d77492492c11606384d5.
Serving (vLLM, ROCm)
Tested with vLLM v0.33.0 (vllm/vllm:v0.33.0-rocm, ROCm 10.0) on 2 × MI210. Quantization is detected from the config; do not pass
--quantization. Pass --dtype=float16: on the MI210 the dense GEMM kernels are 82% faster in fp16 than in the checkpoint's bf16, which is worth
22% to 25% of output throughput at concurrency 2 and 4.
vllm serve <this repo> \ --tensor-parallel-size=2 --dtype=float16 \ --max-model-len=32768 --gpu-memory-utilization=0.92 --kv-cache-dtype=auto \ --max-num-batched-tokens=2048 --language-model-only \ --limit-mm-per-prompt='{"image":0,"video":0}' \ --reasoning-parser=qwen3 --tool-call-parser=qwen3_coder
Tuned fused-MoE config (optional)
moe-configs/ holds the Triton fused-MoE kernel config for this model's shape (E=256, N=512 per card at tensor-parallel size 2, int4_w4a16),
tuned on two MI210 with vLLM v0.33.0 (Triton 3.8.0): all 12 batch sizes from 1 to 2048, finalists re-timed. vLLM does not read it from the model
repository: download the folder and point VLLM_TUNED_CONFIG_FOLDER at it; the startup log then says Using configuration from ....
hf download aidevco/Qwen3.5-122B-A10B-W4A16-g128 --include 'moe-configs/*' --local-dir .export VLLM_TUNED_CONFIG_FOLDER=$PWD/moe-configs
Serving A/B (default config, tuned, tuned, default; fresh pod per run; 1536 prompt and 128 output tokens; bf16 serving): +3.4% to +3.5% output
tokens per second at concurrency 1, +2.0% to +2.7% at 2, +0.3% at 4 (a tie). Only valid for this shape, card and dtype; a newer vLLM or Triton can
change the kernel, so re-check before relying on it.
Results
Measured on 2 × MI210 (64 GB), vLLM v0.33.0, TP=2, --dtype=float16, tuned MoE config, 1536-token prompts and 128 output tokens:
Table with columns: Concurrency, Output tokens/s, ITL (mean), TTFT (mean)| Concurrency | Output tokens/s | ITL (mean) | TTFT (mean) |
|---|
| 1 | 39.6 | 21.8 ms | 443 ms |
| 2 | 64.0 | 26.2 ms | 667 ms |
| 4 | 94.0 | 33.4 ms | 1,059 ms |
Quality (lm-evaluation-harness, 50 samples per task, thinking on, max_gen_toks=8192): gsm8k 0.96 ± 0.03, minerva_math_algebra 0.86 ± 0.05,
humaneval_chat 1.00, mbpp_chat 0.96 ± 0.03. Fifty samples are a smoke test, not a leaderboard score; no comparison with the bf16 original is claimed.
License and attribution
This is a derivative of Qwen3.5-122B-A10B by the Qwen team, released under the Apache License 2.0, the same license as the original (see LICENSE).
The quantization changes the numerical precision of the weights listed above and nothing else. It is not an official Qwen release, and the Qwen team
has not reviewed or endorsed it. For the model's intended use, safety and limitations, see the
original model card; its guidance applies to this build unchanged.