Qwen3.8-27B-INT8-AutoRound API & Inference Endpoint

Available versions

There are four branches. They vary along two axes: the quantization granularity, and whether the hybrid linear attention projections are quantized at all.

Table with columns: Branch, Granularity, linear_attn, Size
Branch	Granularity	linear_attn	Size
`main`	per-channel	`in_proj_qkv`, `in_proj_z`, `out_proj` are INT8	28.8 GiB
`linear-attn-bf16`	per-channel	all of linear_attn stays BF16	33.9 GiB
`main-gs128`	group_size 128	`in_proj_qkv`, `in_proj_z`, `out_proj` are INT8	29.3 GiB
`linear-attn-bf16-gs128`	group_size 128	all of linear_attn stays BF16	34.3 GiB

All four fit on 2x 24GB with tensor parallelism.

If you want the closest match to the base model, take linear-attn-bf16-gs128. If you want the smallest file, take main. The two gs128 branches score better on mean, median and every reported percentile below p99.9, at a cost of about half a gibibyte, so they are the better default unless you are short on disk or already have the per-channel weights in place.

The per-channel branches are kept rather than replaced. They are what the earlier Qwen3.6-27B quantizations used, so they give a like-for-like reference for anyone comparing across model versions, and linear-attn-bf16 has the mildest worst case of the four by a wide margin.

Quantization details

Some layers are left unquantized, either because quantizing them costs more than it saves or because there is no calibration signal reaching them. The layer selection is identical across all four branches. Only the granularity and the linear attention treatment change.

Table with columns: Branch, INT8 Linear, BF16 Linear
Branch	INT8 Linear	BF16 Linear
`main`, `main-gs128`	407	210
`linear-attn-bf16`, `linear-attn-bf16-gs128`	263	354

The INT8 set covers q_proj / k_proj / v_proj / o_proj of self_attn, gate_proj / up_proj / down_proj of mlp, in_proj_qkv / in_proj_z / out_proj of linear_attn, and the MTP layers. The BF16 set covers in_proj_a / in_proj_b of linear_attn, embed_tokens, lm_head and the whole vision tower.

Notes on the choices.

in_proj_a and in_proj_b are not a judgement call. Their output dimension is 48, which is not divisible by 32, so AutoRound skips them regardless of what the layer config asks for. Since in_proj_b and in_proj_a are fused into a single in_proj_ba by vLLM, both ends of that pair land on BF16 and the fused module stays consistent.
The vision tower is excluded because the calibration set is text only, so no calibration input ever reaches it.
embed_tokens and lm_head are excluded by the usual convention. The vocabulary is 248,320 wide, so a per-channel scale would cover 5,120 elements at a time.
Calibration: iters=250, nsamples=1024, seqlen=2048, batch_size=4, , dataset (256) plus (768). Identical for all four branches.

On why there are two granularities. The per-channel setting was carried over from the earlier Qwen3.6-27B recipe without being reexamined. Revisiting it on this model showed that group_size=128 is better on mean KLD, median, p90, p95, p99 and top-1 agreement, for roughly half a gibibyte of extra scale storage. Whether a difference of that size is noticeable in use is a separate question, and this card does not claim that it is.

A note for anyone reproducing this. The shared-dict aliasing bug in AutoRound is still present in 0.15, it has only moved from compressors/utils.py to compressors/layer_config/resolver.py. The regex expansion assigns the same dict object to every matched layer, and the later shape check that forces bits=16 on layers whose weights are not divisible by 32 mutates that shared dict, silently dragging every sibling layer to 16 bit with it. On this architecture a single linear_attn keyword is enough to trigger it, because the forced 16 bit on in_proj_a and in_proj_b propagates to in_proj_qkv, in_proj_z and out_proj and quietly erases the W8 request. Change

layer_config[match] = val

layer_config[match] = copy.deepcopy(val)

or specify each layer by its full suffix rather than by a shared keyword.

Evaluation Results

1. KL Divergence (KLD)

The output distribution of each quantized model was compared directly to that of the base model across 122,640 scored next-token positions.

KL divergence cannot be compared across models, datasets, or evaluation methods, and is only comparable if all of them are the same. You canNOT compare these KLD values to others' KLD reports!

If other quantized models report a very low / high KLD compared to mine in their READMEs, it is typically due to differences in the evaluation methodology.

Table with columns: Metric @ctx 512, main, linear-attn-bf16, main-gs128, linear-attn-bf16-gs128
Metric @ctx 512	main	linear-attn-bf16	main-gs128	linear-attn-bf16-gs128
Mean KLD	0.00124 ± 0.00002	0.00110 ± 0.00001	0.00101 ± 0.00002	0.000956 ± 0.000021
Median KLD	0.000701	0.000628	0.000553	0.000507
P90 / P95 / P99 KLD	0.00231 / 0.00340 / 0.00825	0.00215 / 0.00306 / 0.00719	0.00201 / 0.00273 / 0.00666

Evaluation set: WikiText-2 (test), 240 sequences x 512 tokens = 122,640 scored next-token positions. Values are in nats, and one shared reference pass is used for all four columns, so they are directly comparable.

Because the scored positions are shared, the branches can be compared by a paired difference rather than through their standard errors alone. Taking main as the baseline, the paired mean difference is -0.000149 for linear-attn-bf16 (t = -6.9), -0.000233 for main-gs128 (t = -9.1) and -0.000289 for linear-attn-bf16-gs128 (t = -11.9).

One row does not follow the pattern. The single worst position is far milder on linear-attn-bf16 (0.882) than on any other branch (1.47 to 1.84), even though that branch is beaten everywhere else. Leaving linear attention in BF16 removes the layers where the largest single-position errors originate, and the two granularities barely differ on that point. If your concern is the worst case rather than the average, that is the branch to look at.

2. How to read these numbers

KLD is not an absolute property of a checkpoint. It is a property of a checkpoint measured on a corpus, and the corpus dominates the result. Text that the base model already predicts confidently leaves little probability mass for quantization to move, so the divergence is small. Text with genuine uncertainty leaves a lot, so it is larger. The same checkpoint can land several times apart on two evaluation sets, and the ordering of two checkpoints can change with it.

Two consequences are worth stating plainly.

A mean KLD taken from one model card cannot be placed next to a mean KLD from another unless the corpus, the scored position count, the teacher checkpoint and the precision path all match. Totals in the low thousands drawn from a single document are the most sensitive case, because the positions are correlated and a predictable document pulls the mean down as context accumulates.
Mean KLD and top-1 agreement move together within one measurement setup. Across two different setups they do not, so a checkpoint can show both a worse KLD and a better top-1 than another purely because it was measured on easier text. A pair of numbers only orders checkpoints that were measured the same way.

The protocol used here is stated in full so that it can be reproduced or disputed.

Teacher: Qwen/Qwen3.8-27B at revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, loaded in BF16 through the Hugging Face path with use_cache=False.
Corpus: WikiText-2 raw test, every row concatenated, tokenized once, first 122,880 tokens reshaped to 240 sequences of 512. The first 511 positions of each sequence are scored, giving 122,640.
Metric: KL(P_teacher || Q_candidate) over the full 248,320 token vocabulary, in nats, with FP32 log_softmax. No sampling and no top-k truncation.
Teacher log-probabilities are cached as FP16 and renormalized in FP32 before the divergence is taken, which removes the systematic offset the FP16 round trip would otherwise add.
All four branches are scored against one shared reference pass, so the columns differ only by the checkpoint.

The reference pass was rebuilt from scratch partway through this work and main was rerun against it, reproducing every reported statistic exactly, so the measurement is deterministic end to end.

If you want another checkpoint added to the table above, open a discussion and I will run it through this same script and post the result, including if it comes out ahead of these.

3. Benchmarks

Not run.

Variants

The two linear-attn-bf16 branches are not separate AutoRound runs. Each takes the corresponding quantized checkpoint and swaps the three quantized linear attention projections back to the original BF16 weights. The rounding of the remaining layers was optimized under the assumption that linear attention is INT8, so those layers end up very slightly mismatched, but the substitution only makes the linear attention exact, which moves the block output error down rather than up.

How to Use

Tested with the existing vLLM image built for Qwen3.6-27B, which resolves this model natively since Qwen3.8-27B reports the same qwen3_5 model type and Qwen3_5ForConditionalGeneration architecture. No image rebuild was needed, and the gs128 branches load through the same path as the per-channel ones.

The following configurations were verified to start and answer a request on 2x RTX 3090 with --tensor-parallel-size 2 and --gpu-memory-utilization 0.97.

Table with columns: Branch, Configuration, Result, Weights per rank, KV cache
Branch	Configuration	Result	Weights per rank	KV cache
main	`--language-model-only`	starts	14.13 GiB	240,688 tokens
main	vision enabled	starts	14.57 GiB	217,168 tokens
main	MTP speculative decoding	starts	14.36 GiB

Measured with --max-model-len auto, which resolves to the model's full 262,144 context, and --max-num-seqs 1. The MTP rows use {"method": "mtp", "num_speculative_tokens": 3}.

Two things are worth knowing before you size a deployment around these.

The pool is smaller than the context window on every configuration here, so a single request cannot reach 262,144 tokens on 2x 24GB. The main branch with the language model only comes closest at 240,688. If you need the full window, you need more memory rather than a different branch.
Setting --max-model-len to a small value does not leave the rest of the pool for other requests, it shrinks the pool. At --max-model-len 8192 the same MTP configurations reported roughly half these figures. If you measure this yourself and get a much lower number, check that setting first.

Acknowledgements

Alibaba / Qwen team for the base Qwen3.8-27B model
Intel AutoRound team for the quantization framework
vLLM project for the inference engine and Qwen3_5 MTP support

Available versions

There are four branches. They vary along two axes: the quantization granularity, and whether the hybrid linear attention projections are quantized at all.

Table with columns: Branch, Granularity, linear_attn, Size
Branch	Granularity	linear_attn	Size
`main`	per-channel	`in_proj_qkv`, `in_proj_z`, `out_proj` are INT8	28.8 GiB
`linear-attn-bf16`	per-channel	all of linear_attn stays BF16	33.9 GiB
`main-gs128`	group_size 128	`in_proj_qkv`, `in_proj_z`, `out_proj` are INT8	29.3 GiB
`linear-attn-bf16-gs128`	group_size 128	all of linear_attn stays BF16	34.3 GiB

All four fit on 2x 24GB with tensor parallelism.

Quantization details

Table with columns: Branch, INT8 Linear, BF16 Linear
Branch	INT8 Linear	BF16 Linear
`main`, `main-gs128`	407	210
`linear-attn-bf16`, `linear-attn-bf16-gs128`	263	354

Notes on the choices.

in_proj_a and in_proj_b are not a judgement call. Their output dimension is 48, which is not divisible by 32, so AutoRound skips them regardless of what the layer config asks for. Since in_proj_b and in_proj_a are fused into a single in_proj_ba by vLLM, both ends of that pair land on BF16 and the fused module stays consistent.
The vision tower is excluded because the calibration set is text only, so no calibration input ever reaches it.
embed_tokens and lm_head are excluded by the usual convention. The vocabulary is 248,320 wide, so a per-channel scale would cover 5,120 elements at a time.
Calibration: iters=250, nsamples=1024, seqlen=2048, batch_size=4, , dataset (256) plus (768). Identical for all four branches.

layer_config[match] = val

layer_config[match] = copy.deepcopy(val)

or specify each layer by its full suffix rather than by a shared keyword.

Evaluation Results

1. KL Divergence (KLD)

The output distribution of each quantized model was compared directly to that of the base model across 122,640 scored next-token positions.

KL divergence cannot be compared across models, datasets, or evaluation methods, and is only comparable if all of them are the same. You canNOT compare these KLD values to others' KLD reports!

If other quantized models report a very low / high KLD compared to mine in their READMEs, it is typically due to differences in the evaluation methodology.

Table with columns: Metric @ctx 512, main, linear-attn-bf16, main-gs128, linear-attn-bf16-gs128
Metric @ctx 512	main	linear-attn-bf16	main-gs128	linear-attn-bf16-gs128
Mean KLD	0.00124 ± 0.00002	0.00110 ± 0.00001	0.00101 ± 0.00002	0.000956 ± 0.000021
Median KLD	0.000701	0.000628	0.000553	0.000507
P90 / P95 / P99 KLD	0.00231 / 0.00340 / 0.00825	0.00215 / 0.00306 / 0.00719	0.00201 / 0.00273 / 0.00666

2. How to read these numbers

Two consequences are worth stating plainly.

A mean KLD taken from one model card cannot be placed next to a mean KLD from another unless the corpus, the scored position count, the teacher checkpoint and the precision path all match. Totals in the low thousands drawn from a single document are the most sensitive case, because the positions are correlated and a predictable document pulls the mean down as context accumulates.
Mean KLD and top-1 agreement move together within one measurement setup. Across two different setups they do not, so a checkpoint can show both a worse KLD and a better top-1 than another purely because it was measured on easier text. A pair of numbers only orders checkpoints that were measured the same way.

The protocol used here is stated in full so that it can be reproduced or disputed.

Teacher: Qwen/Qwen3.8-27B at revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, loaded in BF16 through the Hugging Face path with use_cache=False.
Corpus: WikiText-2 raw test, every row concatenated, tokenized once, first 122,880 tokens reshaped to 240 sequences of 512. The first 511 positions of each sequence are scored, giving 122,640.
Metric: KL(P_teacher || Q_candidate) over the full 248,320 token vocabulary, in nats, with FP32 log_softmax. No sampling and no top-k truncation.
Teacher log-probabilities are cached as FP16 and renormalized in FP32 before the divergence is taken, which removes the systematic offset the FP16 round trip would otherwise add.
All four branches are scored against one shared reference pass, so the columns differ only by the checkpoint.

The reference pass was rebuilt from scratch partway through this work and main was rerun against it, reproducing every reported statistic exactly, so the measurement is deterministic end to end.

If you want another checkpoint added to the table above, open a discussion and I will run it through this same script and post the result, including if it comes out ahead of these.

3. Benchmarks

Not run.

Variants

How to Use

The following configurations were verified to start and answer a request on 2x RTX 3090 with --tensor-parallel-size 2 and --gpu-memory-utilization 0.97.

Table with columns: Branch, Configuration, Result, Weights per rank, KV cache
Branch	Configuration	Result	Weights per rank	KV cache
main	`--language-model-only`	starts	14.13 GiB	240,688 tokens
main	vision enabled	starts	14.57 GiB	217,168 tokens
main	MTP speculative decoding	starts	14.36 GiB

Measured with --max-model-len auto, which resolves to the model's full 262,144 context, and --max-num-seqs 1. The MTP rows use {"method": "mtp", "num_speculative_tokens": 3}.

Two things are worth knowing before you size a deployment around these.

The pool is smaller than the context window on every configuration here, so a single request cannot reach 262,144 tokens on 2x 24GB. The main branch with the language model only comes closest at 240,688. If you need the full window, you need more memory rather than a different branch.
Setting --max-model-len to a small value does not leave the rest of the pool for other requests, it shrinks the pool. At --max-model-len 8192 the same MTP configurations reported roughly half these figures. If you measure this yourself and get a much lower number, check that setting first.

Acknowledgements

Alibaba / Qwen team for the base Qwen3.8-27B model
Intel AutoRound team for the quantization framework
vLLM project for the inference engine and Qwen3_5 MTP support

Qwen3.8-27B-INT8-AutoRound

README

Available versions

Quantization details

Evaluation Results

1. KL Divergence (KLD)

KL divergence cannot be compared across models, datasets, or evaluation methods, and is only comparable if all of them are the same. You canNOT compare these KLD values to others' KLD reports!

If other quantized models report a very low / high KLD compared to mine in their READMEs, it is typically due to differences in the evaluation methodology.

2. How to read these numbers

3. Benchmarks

Variants

How to Use

Acknowledgements

Explore FriendliAI today

README

Available versions

Quantization details

Evaluation Results

1. KL Divergence (KLD)

KL divergence cannot be compared across models, datasets, or evaluation methods, and is only comparable if all of them are the same. You canNOT compare these KLD values to others' KLD reports!

If other quantized models report a very low / high KLD compared to mine in their READMEs, it is typically due to differences in the evaluation methodology.

2. How to read these numbers

3. Benchmarks

Variants

How to Use

Acknowledgements