Quantization
Table with columns: field, value| field | value |
|---|
| tool | auto-round 0.14.2 (torch 2.12.0+xpu, transformers 5.8.0) |
| scheme | W4A16, int, symmetric, group size 128 |
| tuning | AutoRound defaults: iters 200, nsamples 128, seqlen 2048, NeelNanda/pile-10k; batch 4 x grad-accum 2, low_gpu_mem_usage |
| packing | auto_round:auto_gptq (quant_method: auto-round) |
| kept bf16 | GDN in_proj_a / in_proj_b (48 layers each), mtp.fc, vision tower, embeddings, lm_head |
| MTP layers | int4, RTN from the source weights (auto-round's missing_tensors path, as in the recipe reference) |
| hardware | Intel Arc Pro B70, 302 min of tuning |
| size | 19.0 GB, 27.8B parameters (the Hub's "6B params" badge counts packed I32 storage: 8 int4 weights per entry) |
MANIFEST.json records the pins and tensor counts. recipe/bake-autoround.sh
is the exact script that produced these weights. It refuses any other source
revision, and before saving it checks the produced quantization_config
against the reference.
Results on an Intel Arc Pro B70
Image: intel/llm-scaler-vllm:0.26.0-b2 plus local patches (see "Serving"),
--quantization inc, --dtype float16, MTP speculative decoding with 3
tokens over a 32k-token draft vocabulary shortlist, fp8 KV cache, 131,072
context, prefix caching, max 16 sequences. Compared with the other auto_round:auto_gptq
int4 of the same source,
JC1DA/…-INT4-W4A16
@ 3b19cdc9 (iters 512, seqlen 4096, MTP kept bf16), on the same stack on the
same day. Both quants come from the same source weights (sha256 of 9 tensors
both keep bf16 match).
Table with columns: this quant, JC1DA | this quant | JC1DA |
|---|
| KV cache pool | 197,982 tokens | 183,317 tokens |
| per-request tok/s, 1 / 4 / 16 concurrent (300-token answers) | 72.2 / 55.5 / 33.8 | 67.5 / 54.4 / 33.3 |
| time to first token, 4k / 32k / 64k prompt | 1.72 / 18.0 / 46.7 s | 1.77 / 18.2 / 50.2 s |
| MTP acceptance (cumulative) | 0.70 | 0.72 |
| internal 50-task eval (JSON, tools, code, reasoning), default effort | 50/50 | 50/50, twice |
internal 50-task eval, reasoning_effort: low | 49/50, then 50/50 | 49/50 |
| NaN-logit and degeneration probes, 100k needle, 6 concurrent JSON-mode requests | pass | pass |
| 90 min sustained-load soak | 750 requests, 0 errors | not run |
Quality is a tie. The speed difference is small but shows on every cell, most
likely because the int4 MTP head is lighter than a bf16 one (not isolated).
The internal eval is a regression check, not a benchmark; run your own evals
for anything that matters.
Serving
vLLM, any loader that reads auto_round:auto_gptq int4. The numbers above
come from intel/llm-scaler-vllm:0.26.0-b2 with patches applied: upstream
vLLM PR 50501 (W4A8 prefill), llm-scaler PR 700, and two local patches, one
that runs inc AutoRound int4 on llm-scaler's sym_int4 kernels and one that
restricts the MTP drafter to a 32k-token vocabulary shortlist. A stock image is
untested with this checkpoint.
The core flags we serve with (bf16 activations fail on the B70, so
--dtype float16):
vllm serve <this repo> --quantization inc --dtype float16 --max-model-len 131072 \ --max-num-seqs 16 --block-size 64 --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --kv-cache-dtype fp8 --mamba-ssm-cache-dtype float16 \ --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder
The chat template and generation config are the source's: temperature 1.0,
top_k 20, top_p 0.95; the template takes reasoning_effort xhigh (default),
medium or low.
Credits
All model quality is DavidAU's fine-tune of Qwen/Qwen3.8-27B; see the
source model card.
The recipe is Frozenlock's. Apache-2.0, as the source.