Quantization
Table | |
|---|
| Method | AWQ, W4A16, group size 128 |
| Format | compressed-tensors (pack-quantized) |
| Ignored | lm_head, MoE router (mlp.gate) |
| Calibration | 64 samples of HuggingFaceH4/ultrachat_200k, seq len 1024 |
| MoE | moe_calibrate_all_experts=True — every calibration token is routed through all 128 experts, so each expert is calibrated |
Qwen3-Coder-30B-A3B is a Mixture-of-Experts model (128 experts, 8 active per token) using
standard GQA attention — so it serves on consumer Ada GPUs with the stock attention
backends, no special kernels required.
Serving with vLLM
vllm serve dark-side-of-the-code/Qwen3-Coder-30B-A3B-Instruct-AWQ \
--max-model-len 65536 \
--kv-cache-dtype fp8 \
--gpu-memory-utilization 0.95 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--override-generation-config '{"temperature":0.7,"top_p":0.8,"top_k":20}'
Notes:
- At
--max-model-len 65536 on a 24 GB card the KV pool holds ~118K tokens (~1.8x
concurrency). The base model's native context is 262K; 64K is the practical cap at 24 GB.
- Tool calling works via vLLM's first-class
qwen3_coder parser.
- Non-thinking model — no reasoning parser needed.
- Pass
--override-generation-config if your client doesn't send sampling params, or vLLM
will default to temperature=1.0, which is far too hot for coding.
Validation
Verified after quantization: correct code generation, working tool_calls through the
qwen3_coder parser, and coherent prose output.
Selected as the local agentic coding model in a head-to-head evaluation driven by the
Pi coding agent, where it reliably executed a read/edit/test tool loop.