Qwen3.5-0.8B-W8A8

Quick start

bash
Copy code
pip install "vllm>=0.17"
huggingface-cli download vadery/Qwen3.5-0.8B-W8A8 --local-dir ./Qwen3.5-0.8B-W8A8

vllm serve ./Qwen3.5-0.8B-W8A8 \
  --max-model-len 32768 \
  --dtype bfloat16 \
  --trust-remote-code \
  --reasoning-parser qwen3 \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'

No additional patches required — both config.json.quantization_config.ignore (covers MTP Linear modules) and actorder field are already fixed.

Performance (single H200 SXM, single-stream)

3-4× faster than the BF16 source running the same MTP recipe.

Architecture preserved

Quantization recipe

python
Copy code
SmoothQuantModifier(smoothing_strength=0.8, mappings=SQ_MAPPINGS,
                    ignore=[...vision, mtp, linear_attn, embed, lm_head...])
GPTQModifier(targets="Linear", scheme="W8A8",
             ignore=[same as above],
             dampening_frac=0.01)

SmoothQuant mappings explicitly cover only the 6 full-attention layers (indices 3, 7, 11, 15, 19, 23 out of 24) plus MLP on every layer — to avoid SmoothQuant trying to fuse into the linear_attn projections which have a non-standard shape.

Calibration: 256 samples × 2048 tokens from HuggingFaceH4/ultrachat_200k.

File size

Notes / gotchas

• vLLM ≥ 0.17 required (qwen3_5_mtp speculative method only landed there).
• transformers ≥ 5.x is required for qwen3_5 model_type.
• The MTP head weights are stored as mtp.* keys in the safetensors file; do not delete or re-quantize them. The companion quantization_config.ignore list explicitly excludes the 8 MTP Linear modules so vLLM treats them as float.
• For multi-stream serving raise --max-model-len and --max-num-seqs to taste.

Reproducing

The quantization script is at https://huggingface.co/vadery/qwen36-27b-ft-grm-w8a8 (sibling 27B model), parameterized for the 0.8B's layer count.

License

Inherits Apache 2.0 from the base model.