letechlead

MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound

Quantization

Table

scheme	W8A16 — 8-bit weights, 16-bit activations, weight-only
group size	128, symmetric
packing	`auto_round:auto_gptq` (GPTQ int32 layout, pack factor 4)
calibration	`NeelNanda/pile-10k`, 128 samples × 512 tokens, batch 1
quantized scope	`model.language_model.layers` — 200 linears
artifact	12.89 GB, 8 shards, 1,160 tensors

Kept in BF16 on purpose:

Vision tower (model.visual.*, 333 tensors) — AutoRound's inference backend does not quantize visual blocks, so the whole tower stays BF16 and is declared bits: 16 in quantization_config.extra_config.
GDN gating projections linear_attn.in_proj_a / in_proj_b (48 projections) — 32-wide outputs cannot be Marlin-repacked (size_n % 64 != 0), so they are stored as plain BF16 weights and marked bits: 16.
Embeddings / lm_head / norms / biases — standard practice, and required by the same extra_config entries.

Quantized: the MLP (gate_proj / up_proj / down_proj), the linear-attention projections (in_proj_qkv, in_proj_z, out_proj) and the full-attention projections (q_proj, k_proj, v_proj, o_proj).

Serving

SGLang v0.5.20 (built from source, commit 94602c9c2b7cbdb8efd5c52802dac6a1c180089e), 2× RTX 3090, tensor parallel 2:

bash

python3 -m sglang.launch_server \  --model-path letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound \  --served-model-name MiMo-V2.6-Distill-Qwen-9B-INT8-AutoRound \  --tp-size 2 \  --reasoning-parser mimo \  --tool-call-parser mimo \  --context-length 262144 \  --mem-fraction-static 0.85 \  --kv-cache-dtype fp8_e5m2

--reasoning-parser mimo is required for the model's reasoning output; the source card also recommends --tool-call-parser mimo for tool use. Verified on SM86 (RTX 3090): the artifact loads through SGLang's auto_round → GPTQ-Marlin int8 path without patching, and answers correctly.

Long context on 2× RTX 3090

The model's native window is 262,144 tokens with no rope scaling, and the whole window runs on the pair:

Only 8 of 32 layers carry a KV cache; the other 24 are GatedDeltaNet linear attention with constant-size state. KV is therefore 32 KiB/token at fp16 and 16 KiB/token with the 8-bit (fp8_e5m2) cache.
Measured pool: 933,890 tokens (3.6× the model max), max_req_input_len 262,138.
Weights + cache fit at ~22.2 GiB per card with --mem-fraction-static 0.85.
Prefill is chunked at 16,384 tokens so a long prompt does not stall the scheduler.
The GDN state cache caps concurrent requests at roughly 59; the window is limited by linear-attention state, not by the KV pool.

Quality (measured, not assumed)

Checkpoint-level coherence against the BF16 source — 8,192 tokens of wikitext-2-raw-v1 test, 15 windows of 1,024 with stride 512, 15,345 scored positions, float32 logits, identical token ids for both models:

Table with columns: metric, BF16, INT8, INT4 (for reference)
metric	BF16	INT8	INT4 (for reference)
perplexity	12.1721	12.1258 (−0.381%)	12.337 (+1.36%)
top-1 agreement vs BF16	—	98.012% (15,040 / 15,345)	87.46% (13,421 / 15,345)
mean KL(BF16 ‖ quantized)	—	0.004067	0.0842

Per-window agreement 97.3–98.8%, per-window KL 0.0012–0.0136. The negative perplexity delta is within noise: at 8-bit the quantization is effectively invisible on this corpus, and it is 20× closer to the BF16 distribution than INT4 (KL 0.004 vs 0.084).

Behavioural probe (7 prompts: arithmetic, factual recall, letter counting, sequence completion, two instruction-following constraints, one open prose question), graded against the same prompts on the BF16 source:

Table with columns: model, score
model	score
BF16 source	7/7
INT8 W8A16	7/7
INT4 W4A16	5/7

INT4 misses the bat-and-ball trap and an "exactly two sentences" constraint; INT8 gets both.

Throughput

llama-benchy 0.3.5, pp 4096 / tg 512, 3 runs each, cold, concurrency 1, exact tokenizer, 2× RTX 3090 TP=2, 262,144 ctx with the 8-bit KV cache:

Table with columns: depth, pp tok/s, tg tok/s, peak tg, TTFT, E2E est.
depth	pp tok/s	tg tok/s	peak tg	TTFT	E2E est.
8,000	2367.7	91.94	94.3	5.12 s	10.69 s
16,000	2397.8	95.40	96.7	8.39 s	13.75 s
32,000	2370.1	93.98

All 7 depths × 3 runs completed, no failures. The 257,000 rung is a 261,096-token prompt.

Against INT4 at the same configuration:

Table with columns: depth, INT4 tg, INT8 tg, Δ
depth	INT4 tg	INT8 tg	Δ
8K	123.01	91.94	−25.3%
16K	122.27	95.40	−22.0%
32K	120.01	93.98	−21.7%
58K	115.41	86.93	−24.7%
128K

Decode costs 17–25% because decode is weight-bandwidth-bound and int8 doubles the weight bytes; prefill and TTFT are essentially unchanged (1746 vs 1766 tok/s at 257K) because prefill is compute-bound.

Which one to use

INT8 (this artifact) — when quantization must not change model behaviour, or when 12.89 GB fits where 18.85 GB BF16 does not (a single 24 GB card leaves ~11 GB for cache instead of ~5 GB; 16 GB cards and CPU-offload setups cannot hold BF16 at all).
INT4 W4A16 — when you want maximum decode speed and a smaller artifact, and can accept the measured quality delta (+1.36% perplexity, 87.5% top-1 agreement, 5/7 on the probe).
BF16 — when the pair has room and you want the exact source distribution.

Verification performed

760/760 source tensors accounted for; 0 missing, 0 extra, 1,160 tensors total.
Every quantized linear has its full qweight / qzeros / scales triple.
No quantized output width fails % 64; no quantized output is narrower than 64.
Group packing aligned (in-features divisible by 128/pack-factor).
Loads and serves in SGLang v0.5.20 on SM86 with the GPTQ-Marlin int8 path; smoke test correct.
Tokenizer identity: tokenizer.json and chat_template.jinja are byte-identical to the BF16 source, and the served tokenizer returns the same ids as both the artifact's own tokenizer and the source for identical inputs via /tokenize (English, Arabic, code). tokenizer_config.json differs from the source in one field only: tokenizer_class is as written by AutoRound, where the source has .

Files

text

model-0000{1..8}-of-00008.safetensors   int8 weights + BF16 kept tensorsmodel.safetensors.index.json            1160 tensors, 8 shardsconfig.json                             model config + embedded quantization_configquantization_config.json                bits 8, g128, sym, auto_round:auto_gptq, extra_configtokenizer.json / tokenizer_config.json / chat_template.jinjapreprocessor_config.json / processor_config.json / generation_config.json

Attribution

Base model: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (MIT). Quantization: AutoRound 0.14.2. Serving stack: SGLang v0.5.20. Quantized and verified by LeTechLead.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

letechlead

Model Tree

Base

XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Quantization

Table

scheme	W8A16 — 8-bit weights, 16-bit activations, weight-only
group size	128, symmetric
packing	`auto_round:auto_gptq` (GPTQ int32 layout, pack factor 4)
calibration	`NeelNanda/pile-10k`, 128 samples × 512 tokens, batch 1
quantized scope	`model.language_model.layers` — 200 linears
artifact	12.89 GB, 8 shards, 1,160 tensors

Kept in BF16 on purpose:

Vision tower (model.visual.*, 333 tensors) — AutoRound's inference backend does not quantize visual blocks, so the whole tower stays BF16 and is declared bits: 16 in quantization_config.extra_config.
GDN gating projections linear_attn.in_proj_a / in_proj_b (48 projections) — 32-wide outputs cannot be Marlin-repacked (size_n % 64 != 0), so they are stored as plain BF16 weights and marked bits: 16.
Embeddings / lm_head / norms / biases — standard practice, and required by the same extra_config entries.

Serving

SGLang v0.5.20 (built from source, commit 94602c9c2b7cbdb8efd5c52802dac6a1c180089e), 2× RTX 3090, tensor parallel 2:

bash

python3 -m sglang.launch_server \  --model-path letechlead/MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound \  --served-model-name MiMo-V2.6-Distill-Qwen-9B-INT8-AutoRound \  --tp-size 2 \  --reasoning-parser mimo \  --tool-call-parser mimo \  --context-length 262144 \  --mem-fraction-static 0.85 \  --kv-cache-dtype fp8_e5m2

Long context on 2× RTX 3090

The model's native window is 262,144 tokens with no rope scaling, and the whole window runs on the pair:

Only 8 of 32 layers carry a KV cache; the other 24 are GatedDeltaNet linear attention with constant-size state. KV is therefore 32 KiB/token at fp16 and 16 KiB/token with the 8-bit (fp8_e5m2) cache.
Measured pool: 933,890 tokens (3.6× the model max), max_req_input_len 262,138.
Weights + cache fit at ~22.2 GiB per card with --mem-fraction-static 0.85.
Prefill is chunked at 16,384 tokens so a long prompt does not stall the scheduler.
The GDN state cache caps concurrent requests at roughly 59; the window is limited by linear-attention state, not by the KV pool.

Quality (measured, not assumed)

Table with columns: metric, BF16, INT8, INT4 (for reference)
metric	BF16	INT8	INT4 (for reference)
perplexity	12.1721	12.1258 (−0.381%)	12.337 (+1.36%)
top-1 agreement vs BF16	—	98.012% (15,040 / 15,345)	87.46% (13,421 / 15,345)
mean KL(BF16 ‖ quantized)	—	0.004067	0.0842

Table with columns: model, score
model	score
BF16 source	7/7
INT8 W8A16	7/7
INT4 W4A16	5/7

INT4 misses the bat-and-ball trap and an "exactly two sentences" constraint; INT8 gets both.

Throughput

llama-benchy 0.3.5, pp 4096 / tg 512, 3 runs each, cold, concurrency 1, exact tokenizer, 2× RTX 3090 TP=2, 262,144 ctx with the 8-bit KV cache:

Table with columns: depth, pp tok/s, tg tok/s, peak tg, TTFT, E2E est.
depth	pp tok/s	tg tok/s	peak tg	TTFT	E2E est.
8,000	2367.7	91.94	94.3	5.12 s	10.69 s
16,000	2397.8	95.40	96.7	8.39 s	13.75 s
32,000	2370.1	93.98

All 7 depths × 3 runs completed, no failures. The 257,000 rung is a 261,096-token prompt.

Against INT4 at the same configuration:

Table with columns: depth, INT4 tg, INT8 tg, Δ
depth	INT4 tg	INT8 tg	Δ
8K	123.01	91.94	−25.3%
16K	122.27	95.40	−22.0%
32K	120.01	93.98	−21.7%
58K	115.41	86.93	−24.7%
128K

Which one to use

INT8 (this artifact) — when quantization must not change model behaviour, or when 12.89 GB fits where 18.85 GB BF16 does not (a single 24 GB card leaves ~11 GB for cache instead of ~5 GB; 16 GB cards and CPU-offload setups cannot hold BF16 at all).
INT4 W4A16 — when you want maximum decode speed and a smaller artifact, and can accept the measured quality delta (+1.36% perplexity, 87.5% top-1 agreement, 5/7 on the probe).
BF16 — when the pair has room and you want the exact source distribution.

Verification performed

760/760 source tensors accounted for; 0 missing, 0 extra, 1,160 tensors total.
Every quantized linear has its full qweight / qzeros / scales triple.
No quantized output width fails % 64; no quantized output is narrower than 64.
Group packing aligned (in-features divisible by 128/pack-factor).
Loads and serves in SGLang v0.5.20 on SM86 with the GPTQ-Marlin int8 path; smoke test correct.
Tokenizer identity: tokenizer.json and chat_template.jinja are byte-identical to the BF16 source, and the served tokenizer returns the same ids as both the artifact's own tokenizer and the source for identical inputs via /tokenize (English, Arabic, code). tokenizer_config.json differs from the source in one field only: tokenizer_class is as written by AutoRound, where the source has .

Files

text

model-0000{1..8}-of-00008.safetensors   int8 weights + BF16 kept tensorsmodel.safetensors.index.json            1160 tensors, 8 shardsconfig.json                             model config + embedded quantization_configquantization_config.json                bits 8, g128, sym, auto_round:auto_gptq, extra_configtokenizer.json / tokenizer_config.json / chat_template.jinjapreprocessor_config.json / processor_config.json / generation_config.json

Attribution

Base model: XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B (MIT). Quantization: AutoRound 0.14.2. Serving stack: SGLang v0.5.20. Quantized and verified by LeTechLead.

MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound

README

Quantization

Serving

Long context on 2× RTX 3090

Quality (measured, not assumed)

Throughput

Which one to use

Verification performed

Files

Attribution

Explore FriendliAI today

README

Quantization

Serving

Long context on 2× RTX 3090

Quality (measured, not assumed)

Throughput

Which one to use

Verification performed

Files

Attribution