Model Details
Qwen3.8-27B (qwen3_5 / Qwen3_5ForConditionalGeneration):
- Type: Vision-language model (hybrid linear + full attention language model + vision encoder)
- Parameters: 27.79B total
- Layers: 64 — 48 linear-attention (GatedDeltaNet) + 16 full-attention (every 4th layer: 0, 4, 8, …, 60)
- Hidden size: 5,120
- Intermediate size: 17,408
- Attention heads: 24 × 256 dim (full-attn); 16 key × 128 dim / 128 value head (linear-attn)
- Vocabulary: 248,320
- Activation: SiLU (SwiGLU MLP,
output_gate_type=swish)
- Context: 262,144 tokens (max_position_embeddings)
- Vision encoder: 27-layer ViT, hidden size 1,152, patch size 16, 16 heads
- MTP: 1 layer,
num_speculative_tokens=3, shared embeddings
Each linear-attention layer has in_proj_qkv, in_proj_z, out_proj plus recurrent
state paths in_proj_a / in_proj_b, a norm, and a conv1d (kernel dim 4). Each
full-attention layer has standard q_proj / k_proj / v_proj / o_proj
(4 KV heads, partial rotary 0.25). All layers share gate_proj / up_proj / down_proj
MLP blocks. The vision tower (model.visual.*) processes image/video patches and injects
embeddings into the language model via image/video token boundaries.
Quantization Configuration
Precision Assignment
Table with columns: Module group, Format, Weights, Activations, Layers| Module group | Format | Weights | Activations | Layers |
|---|
| MLP (gate/up/down) — early | INT4 W4A16, g64, sym | 4-bit int, group, memoryless_minmax | BF16 (not quantized) | 0–31 |
| MLP (gate/up/down) — late | INT5 W4A16, g64, sym | 5-bit int, group, memoryless_minmax | BF16 (not quantized) | 32–63 |
Weight-only: input_activations: null, output_activations: null for all groups.
Weights are dequantized to BF16 at the kernel boundary — no activation quantization error.
KV cache is left in BF16 (no kv_cache_scheme).
Modules Kept in BF16
Table with columns: Pattern, Reason| Pattern | Reason |
|---|
re:.*visual.* | Vision tower — small, quantization-sensitive |
re:.*mtp.* | MTP head — kept BF16 for speculative decoding |
lm_head | Output projection — kept BF16 for precision |
model.language_model.embed_tokens | Token embeddings — kept BF16 |
model.language_model.layers.*.linear_attn.in_proj_a / in_proj_b | Recurrent state path — non-power-of-64 dims |
Calibration
- Tool: AutoRound 0.14.2 (MLLM block-walker path)
- Iters: 200
- Source:
malaiwah/qwen38-27b-fidelity-suite-v3 tokens (181 contexts × 2048 tokens)
- Samples: 181, max sequence length 2048
- Format: Decoded token IDs from the fidelity suite, split 2/3 user / 1/3 assistant
- Scheme base:
W4A16, overridden per-module by an explicit layer_config (exact
module names — AutoRound 0.14.2 does not expand regex keys for the MLLM path)
- Seed: 0
Build command
python scripts/quant_pessoa_autoround_v7.py \
--model /models/Qwen3.8-27B \
--output /models/Qwen3.8-27B-Pessoa-AR11-w6-late5-32 \
--iters 200 --calib-samples 181 --calib-max-len 2048 \
--lat-bits 6 --sat-bits 6 \
--late-mlp-bits 5 --late-mlp-start 32 \
--base-mtp-dir /models/Qwen3.8-27B
Fidelity Evaluation (KLD)
Distribution-fidelity measured against the BF16 base on the
qwen38-27b-fidelity-suite-v3 capture-replay harness: 136 contexts, 278,392 scored
positions, comparing the full next-token softmax distribution (KL divergence in nats,
float64 accumulation, two-pass) using the shared BF16 lm_head.
Headline Metrics
Table with columns: Metric, Value| Metric | Value |
|---|
| Mean KLD | 0.003971 nats |
| Median KLD | 0.001230 nats |
| p95 KLD | 0.01436 nats |
| p99 KLD | 0.05407 nats |
| p999 KLD | 0.1941 nats |
| Max KLD | 1.5037 nats |
| Top-1 agreement | 97.64% |
| Mean JSD | 0.001408 bits |
| Contexts |
Bootstrap CI (mean KLD, 10,000 resamples, 41 clusters)
Table with columns: Value | Value |
|---|
| Mean | 0.003971 |
| 95% CI low | 0.003176 |
| 95% CI high | 0.004916 |
KLD by Stratum
Table with columns: Stratum, Contexts, Mean KLD| Stratum | Contexts | Mean KLD |
|---|
| Multilingual | 7 | 0.001888 |
| Scientific | 39 | 0.002028 |
| Encyclopedic | 13 | 0.003125 |
| Code | 36 | 0.003856 |
| Literary | 41 | 0.006545 |
As is typical for weight-only quantization of this family, the literary stratum
(Project Gutenberg prose) carries the highest mean KLD, while scientific and
multilingual contexts are the closest to the base distribution.
Worst Contexts
Table with columns: #, Stratum, Cluster, Mean KLD, Max KLD, Top-1| # | Stratum | Cluster | Mean KLD | Max KLD | Top-1 |
|---|
| 156 | literary | pg2701 (Moby-Dick) | 0.01769 | 1.5037 | 97.07% |
| 79 | literary | pg174 (Scrooge) | 0.01710 | 0.4504 | 95.55% |
| 51 | literary | pg98 (Tale of Two Cities) |
The worst individual token (max KLD 1.50 nat) occurs in a Moby-Dick passage; even there
the top-1 agreement remains above 97%.
Comparator
- Vocab chunk: 24,832 · Device: CUDA · Accumulation: float64 · Two-pass: true
AutoRound Ladder (Pessoa series)
AR11 sits in a sweep of AutoRound mixed-precision variants, all with INT6 attention
(lat + sat) and a progressively later INT4→INT5 MLP boundary:
Table with columns: Variant, MLP early / late split, MLP late bits, Mean KLD, Top-1, Size| Variant | MLP early / late split | MLP late bits | Mean KLD | Top-1 | Size |
|---|
| AR6-INT4 | all INT4 | 4 | 0.009011 | 96.54% | — |
| AR7-w6 | no late split | — | 0.008491 | 96.58% | — |
| AR8-w6-late5 | ≥8 |
KLD falls monotonically as the INT4→INT5 boundary moves earlier (more layers at INT5).
AR11's 32/32 split is the knee of the curve: each step beyond it yields diminishing
fidelity gains for the added size, while each step before it costs noticeably more KLD.
Usage
# vLLM serve (RTX 5090 / Blackwell SM120 + FlashInfer)
vllm serve /models/Qwen3.8-27B-Pessoa-AR11-w6-late5-32 \
--served-model-name qwen38-27b-pessoa-5090 \
--max-model-len 131072 \
--max-num-seqs 32 \
--quantization compressed-tensors \
--trust-remote-code \
--attention-backend flashinfer \
--enable-chunked-prefill \
--enable-prefix-caching \
--reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
The endpoint is OpenAI-compatible. MTP speculative decoding is enabled
(num_speculative_tokens=3), providing a decode speedup when acceptance is high.
Hardware & Runtime
Table with columns: Field, Value| Field | Value |
|---|
| Target GPU | NVIDIA RTX 5090 (32 GB) / Blackwell SM120 |
| Checkpoint | 22.59 GB (fits 32 GB with KV cache + MTP headroom) |
| Engine | vLLM, FlashInfer backend |
| MTP speculative decoding | 3 speculative tokens |
| Max model len | 131,072 (recommended serve) |
| Vision | BF16, full image/video understanding retained |
Limitations
- No task-accuracy benchmarks yet. This model has a KLD fidelity report but has not
been run through the GSM8K / MMLU-Pro / GPQA-Diamond accuracy suite. KLD is a strong
proxy for distribution closeness but is not a direct task-score measurement. Validate
on your own workload before production use.
- Quantization is lossy. Weight-only INT4/INT5/INT6 eliminates activation error but
retains weight rounding error. The literary stratum shows the highest divergence.
- Blackwell-optimized. The INT pack-quantized MMA paths are exercised on SM120
(Blackwell). Validate on your hardware (Ada/Hopper SM89/SM90 also support the relevant
integer paths).
- AutoRound 0.14.2 MLLM path quirk. Regex
layer_config keys are not expanded for the
MLLM block-walker; this checkpoint uses explicit per-module keys (400 entries) and a
post-build config guard verifying exported group bits match the recipe.
- RTX 5090 naming. The "5090" in the model name denotes the target memory envelope
(single 32 GB card). The checkpoint is hardware-agnostic compressed-tensors and runs on
any vLLM-supported GPU with sufficient VRAM.
Acknowledgements
Standing on the shoulders of giants. This work would not exist without the
rtx6kpro community and the
broader local-inference-lab Discord — the open protocol, tooling, and baseline
measurements that made systematic quantization comparison possible.
The fidelity suite (malaiwah/qwen38-27b-fidelity-suite-v3) and the KLD capture-replay
harness provided the distribution-level validation. AutoRound
(Intel) produced the optimized rounding.