What is quantized
Table with columns: Component, Precision| Component | Precision |
|---|
LM Linear layers (MLP, full-attn q/k/v/o_proj, linear-attn projections) | INT4 W4A16, g128, symmetric |
| Vision tower + merger/projector | BF16 |
Embeddings, tied lm_head, RMSNorms | BF16 |
MTP head (mtp.*) | BF16 (post-quant splice via save_mtp_tensors_to_checkpoint) |
- 248 Linear layers quantized; on-disk ~5.0 GB (vs ~8 GB BF16).
Quality (AWQ vs BF16)
OpenLLM-lite, n=200 per task, greedy logprob MC scoring on RTX 5060 Ti (sm_120):
Table with columns: Task, BF16, AWQ, Recovery| Task | BF16 | AWQ | Recovery |
|---|
| MMLU | 0.710 | 0.700 | 98.6% |
| ARC-Challenge | 0.545 | 0.555 | 101.8% |
| HellaSwag | 0.705 | 0.670 | 95.0% |
| Winogrande | 0.685 | 0.675 | 98.5% |
| TruthfulQA MC1 |
Mean MC recovery: 98.1%.
Evals
Full AWQ vs BF16 dump: LostGentoo/awq-quant-evals
(qwen35_4b_awq_vs_bf16.json, plus combined quant_quality_evals.json).
Optimal vLLM serve
compressed-tensors is auto-detected - do not pass --quantization awq.
Recommended (text + vision, MTP on)
vllm serve LostGentoo/Qwen3.5-4B-AWQ \
--trust-remote-code \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--limit-mm-per-prompt '{"image":1}' \
--default-chat-template-kwargs '{"enable_thinking": false}' \
--generation-config vllm \
--mamba-cache-mode align \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
Notes:
- MTP: requires
model_mtp.safetensors (included). Older docs may say
qwen3_5_mtp; current vLLM remaps that to mtp.
--mamba-cache-mode align: needed for hybrid Gated-DeltaNet + MTP /
prefix-cache paths (all is unsupported for Qwen3.5 MTP).- Thinking: default Qwen3.5 thinking can burn tokens; keep it off unless you
want CoT. For hard math/coding, set
"enable_thinking": true and raise
max_tokens.
- Vision: keep
--limit-mm-per-prompt '{"image":1}' so the VLM path is
enabled; omit only for text-only deployments.
- Blackwell (sm_120): Marlin W4A16 works; add
--enforce-eager only if
first bring-up hits compile issues.
Python
from vllm import LLM, SamplingParams
llm = LLM(
model="LostGentoo/Qwen3.5-4B-AWQ",
trust_remote_code=True,
max_model_len=8192,
limit_mm_per_prompt={"image": 1},
speculative_config={"method": "mtp", "num_speculative_tokens": 3},
)
sp = SamplingParams(temperature=0.7, top_p=0.8, top_k=20, max_tokens=256)
print(llm.generate(["Explain entropy in one sentence."], sp)[0].outputs[0].text)
OpenAI-compatible client
curl http://127.0.0.1:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "LostGentoo/Qwen3.5-4B-AWQ",
"messages": [{"role":"user","content":"In one sentence, what is entropy?"}],
"max_tokens": 128,
"temperature": 0.7,
"chat_template_kwargs": {"enable_thinking": false}
}'
Recipe
- llm-compressor 0.12:
AWQModifier(duo_scaling=False) + QuantizationModifier(W4A16)
- Ignore:
re:.*visual.*, re:.*lm_head, re:.*mtp.*
- Calib: 256 mixed-modal samples from
lmms-lab/flickr30k, seq 2048
- MTP splice after save from
Qwen/Qwen3.5-4B
License
Apache-2.0, inherited from the base model.