RukaRat

Qwen3.8-27B-INT8-W8A8-imatrix-MTP

README

License: apache-2.0

Why INT8

FP8 needs compute capability 8.9 or higher, so on Ampere cards like the 3090, A6000 and A100 it isn't available and INT8 W8A8 is the practical choice. Worth checking what you're on:

bash
nvidia-smi --query-gpu=compute_cap --format=csv

If you're on Ada or newer, an FP8 build will probably serve you better than this one.

What's actually quantized

I read this off the tensor dtypes rather than trusting the config:

Table with columns: module group, dtypes, size
module group	dtypes	size
MLP	192 INT8 + 192 BF16	15.94 GiB
linear_attn (48 layers)	432 BF16 + 144 INT8	5.21 GiB
norms / misc	130 BF16	2.37 GiB
`lm_head`	BF16	2.37 GiB
full_attn (16 layers)	96 BF16 + 64 INT8	1.56 GiB
vision tower	333 BF16	0.86 GiB
MTP	15 BF16	0.79 GiB

The MLP and full-attention projections carry most of the quantization. The vision tower stays BF16 throughout, and the Gated DeltaNet linear_attn layers are only partly quantized, since in_proj_a, in_proj_b and norm are excluded. That's where most of that 5.21 GiB sits. lm_head and MTP are excluded too.

It's a fairly conservative recipe. You could get this model smaller, though in my experience you tend to give up speculative decoding or vision along the way, and for a two card setup the tradeoff didn't seem worth it.

Running it in vLLM

This is the config I use on the dual 3090s:

bash
vllm serve RukaRat/Qwen3.8-27B-INT8-W8A8-imatrix-MTP \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --gpu-memory-utilization 0.92 \
  --hf-overrides '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":1.5,"original_max_position_embeddings":262144,"mrope_interleaved":true,"mrope_section":[11,11,10],"partial_rotary_factor":0.25,"rope_theta":10000000}}}' \
  --max-model-len 311296 \
  --kv-cache-dtype fp8_e4m3 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --max-num-batched-tokens 4096 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
  --limit-mm-per-prompt '{"image":4,"video":0}' \
  --mm-processor-kwargs '{"max_pixels":2000000,"min_pixels":65536}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --default-chat-template-kwargs '{"reasoning_effort": "low"}'

Set --max-model-len just under the GPU KV cache size that vLLM prints on startup. On a single card or a different pair you'll want to adjust --tensor-parallel-size and the context length to match.

Getting to 311k context on 48 GB

The 311,296 figure is past this model's native 262,144 window, so the --hf-overrides line above isn't optional. It applies YaRN at 1.5x, and without it vLLM will refuse to start because --max-model-len exceeds max_position_embeddings. Keep mrope_section, partial_rotary_factor and rope_theta in that override or the model won't load at all.

Something I didn't expect: extending with YaRN actually made deep-context behaviour better rather than worse for me. At 1.5x scaling, position 250k sits where roughly 167k would land natively, comfortably inside the range the model was trained on, so the far end of the context is less stretched than it would be running native.

Three things together are what make the long context fit:

--kv-cache-dtype fp8_e4m3 roughly halves KV cache memory against fp16. I wouldn't go below 8-bit here, quality drops off noticeably.
--mm-processor-kwargs with the pixel cap. Without it vLLM reserves memory for a worst-case image, and setting it gave me back around 54k tokens of cache.
--max-num-batched-tokens 4096. I tried 8192 and 16384 and both cost me KV cache without giving anything back, though I only ever serve one request at a time so your mileage will differ if you're batching.

That combination reports about 349,800 tokens of KV cache on 2x24 GB, and I set --max-model-len to 311,296 to leave room, since MTP spends cache on the drafter too. Turning speculative decoding off frees roughly 25% more context if you'd rather have length than speed.

The pixel cap matters for vision

The vision tower has 2,304 position embeddings and each merged token covers 1,024 px, which works out to roughly 2.36 MP per image. Without --mm-processor-kwargs, images much above 1536x1536 will fail with:

markdown
Mismatch in `image` token count between text and `input_ids`

Setting it also gives you back a chunk of KV cache, since vLLM otherwise reserves room for a worst-case image.

A note on reasoning_effort

The chat template defaults to xhigh, the maximum. In my testing that spent a lot of tokens on thinking and would sometimes run out of budget before writing an answer. I settled on low as a default and raise it when a task really calls for it. It also works as a per-request field if you'd rather set it per call.

Either way it's worth giving it plenty of max_tokens. This model thinks at length, and if it runs out mid-thought you get reasoning back with empty content.

Sampling

markdown
temperature 0.6 · top_k 20 · top_p 0.86 · repetition_penalty 1.07
presence_penalty 0.0 · frequency_penalty 0.0

I'd avoid temperature 0. Qwen mentions that greedy decoding degrades thinking mode, and that matched what I saw, reasoning that loops with no output at the end.

The recipe

markdown
format      compressed-tensors, W8A8
weights     8-bit int · symmetric · per-channel · static · observer: imatrix-mse
activations 8-bit int · symmetric · per-token · dynamic
targets     Linear
ignore      all model.visual.* blocks · linear_attn in_proj_a / in_proj_b / norm
            · lm_head · re:.*mtp.*

Calibration was 512 sequences of roughly 2,000 tokens each, half of them containing tool calls, built from public open source Python (the vLLM and ComfyUI trees). I'm not shipping the corpus itself, but the description above should be enough to rebuild something equivalent.

imatrix-mse weights the quantization error by how much each activation actually matters, rather than minimizing average weight error uniformly. It needs calibration data and pipeline="sequential". If either is missing, llmcompressor falls back to a data-free pipeline, logs

markdown

imatrix_mse: no importance data available. Falling back to uniform MSE

, and finishes normally while producing something different from what you asked for. Worth checking the log for that line.

Three things to watch for if you rebuild this

Load with AutoModelForImageTextToText. With AutoModelForCausalLM you get the language model on its own and no vision tower, and there's no error to tell you.
llmcompressor drops the MTP module, so graft the 15 mtp.* tensors back from the base checkpoint afterwards. They stay BF16 with no scales, which is expected.
Add re:.*mtp.* to quantization_config.ignore. Without it vLLM reads those BF16 tensors as INT8 and speculative decoding sits at 0% acceptance without complaining, or throws KeyError: 'weight_scale'. After fixing it, clear ~/.cache/vllm/torch_compile_cache, otherwise vLLM reuses the graph it compiled under the broken config and the fix looks like it did nothing. That one cost me an hour.

Numbers

Measured on the dual 3090s at TP=2, no NVLink, vLLM 0.22.0, fp8_e4m3 KV cache. One card sits on an x4 chipset slot, so this isn't a best case setup. Acceptance also depends a lot on workload, so these are a rough guide rather than a promise.

These are all my own builds of the same base model, on the same machine and harness:

Table with columns: build, weight observer, MTP acceptance, tokens/forward-pass
build	weight observer	MTP acceptance	tokens/forward-pass
this one	`imatrix-mse`	62.5-63.5%	2.87-2.89
minmax	`memoryless_minmax`	63.0%	2.92
W8A16	minmax, group-wise	59.2%	2.76
SmoothQuant

imatrix-mse and minmax come out about even once you allow for noise, which ran around 20% on my setup. Both did better than the plain mse observer, which clips the activation outliers the drafter depends on.

At --max-model-len 311296 with the fp8 KV cache I get around 349,800 tokens of KV cache across the two cards.

On speculative decoding

num_speculative_tokens: 3 worked best for me. If you're testing different values of N, it's worth comparing tokens per forward pass rather than acceptance percentage, since fewer draft positions just makes for an easier test rather than more throughput.

Qwen3.8 ships one trained MTP layer, so there's no adding capacity through configuration. Acceptance also seems bounded by the architecture: 48 of the 64 layers are Gated DeltaNet linear attention, and their recurrent state is compressed, so a rejected draft can't be rolled back the way you'd truncate a KV cache suffix. On the same harness Qwen3.6-27B reached 73.4% acceptance and 3.18 tok/pass against roughly 63% and 2.88 here, so about 92% of the older model's speculative throughput. I tried a few things to close it, quantization strategy, draft sampling, N of 2/3/5, mamba state dtypes, KV dtypes, and a vLLM nightly that was meant to address this directly, and none of them made much difference.

Limitations

Built with Ampere in mind, and a dual 3090 setup in particular. On Ada, Hopper or Blackwell you'll likely do better with FP8 or NVFP4.
Not compressed as far as it could be. Vision, lm_head, MTP and a good chunk of linear_attn are still BF16.
I measured throughput and speculative acceptance rather than accuracy, so there's no MMLU or GSM8K comparison against the base model here.

License

Same license as the base model, Qwen/Qwen3.8-27B, which is Apache 2.0. The base repo has the authoritative terms. All the credit goes to the Qwen team, this is just a quant of their work.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider