1. Identity & scope
Table with columns: Property, Value| Property | Value |
|---|
| Base model | Qwen/Qwen3.8-27B (Apache-2.0) |
| Architecture | Qwen3_5ForConditionalGeneration (model_type qwen3_5), image-text-to-text |
| Parameters | 27,781,427,952 (~27.78 B), as reported by the base model |
| Quantization | GPTQ, 4-bit, group size 32, symmetric, non-act-ordered |
| Layers | 64 — 48 Gated-DeltaNet linear-attention + 16 full-attention |
| Native context | 262,144 tokens (verified serving at 163,840) |
| MTP | 1 draft layer, 15 tensors, kept bf16, merged into the main shards |
| License | Apache-2.0 (base and derived) |
| Quantized by | Bernhard Brieger |
| Quantization date | 2026-08-21 |
Intended use. General instruction following, coding, tool/function calling,
and long-context workloads (retrieval over documents >100k tokens) on
hardware that fits ~20 GB of weights. The 4-bit weights use the standard GPTQ
packing format, so any runtime that loads GPTQ checkpoints works.
Out of scope. This is a plain quantization of the base model — no
fine-tuning, instruction distillation, or safety retraining was applied. It is
not intended as a substitute for the base model where full precision is
required, and the numbers below characterize this specific checkpoint, not the
base model in general.
What is quantized vs. kept at bf16. 400 linear modules (all mlp
projections and attention projections across the 64 layers) are quantized to
4-bit: that is 24,326,963,200 of the 27.78 B parameters (87.6 %). The
remaining 3.45 B parameters (12.4 %) stay at bf16: the token-embedding and
lm_head matrices (1.27 B each), the MTP draft layer (425 M), the vision
tower (461 M), and the per-layer layernorms / GDN conv1d / in_proj_a /
in_proj_b / output-gate parameters.
2. Quantization recipe
Self-contained recipe — every parameter below is stored in
quantize_config.json in this repo.
Table with columns: Component, Version| Component | Version |
|---|
| GPTQModel | 7.3.4 |
| transformers | 5.15.0 |
| PyTorch | 2.13.0+cu130 |
| Python | 3.12 |
| GPU | 1× NVIDIA RTX 5090, 32 GB |
| CPU | AMD Ryzen 9 9950X3D, 16C/32T |
| RAM | 64 GB DDR5 |
GPTQ configuration
Table with columns: Parameter, Value, Note| Parameter | Value | Note |
|---|
bits | 4 | |
group_size | 32 | uniform — full vLLM/SGLang compatibility |
sym | true | symmetric quantization |
desc_act | false |
Modules kept at bf16 (the 11 dynamic exclusion patterns)
GPTQ exclusions are negative patterns: a matched module is skipped and left
at bf16. The recipe below is what is stored in quantize_config.json.dynamic.
Each pattern is namespace-agnostic — it matches both the HuggingFace full
module path and vLLM's layer prefix, because both consumers anchor the same
regular expression at position 0.
Table with columns: Pattern, What it protects, Why| Pattern | What it protects | Why |
|---|
*.input_layernorm | input layernorms | precision-critical; not in GPTQModel's supported-module set |
*.post_attention_layernorm | post-attention layernorms | same |
*.self_attn.q_norm | query normalization | same |
*.self_attn.k_norm | key normalization | same |
Why the MTP pattern is written the way it is. vLLM gates MTP on any
dynamic key whose text contains mtp (qwen3_5_mtp.py). The form
(?:.*\.)?mtp\..* satisfies that gate and matches a root mtp.* without
false-positiving on e.g. model.language_model.embed_tokens (no dot before
its "mtp"), unlike a naive -:.*mtp.*.
Calibration corpus
Table with columns: Metric, Value| Metric | Value |
|---|
| Sequences | 2,026 |
| Total tokens | 1,802,645 |
| Longest sequence | 132,968 tokens |
| Random seed | 42 |
| Ordering | desc (longest first) |
| Concat block size | 40,960 (44 full blocks + 1 final; 0 % real-token loss) |
| Calibration device | CPU (per-layer block to GPU) |
Built from a curated pool of public, non-gated sources (see
License & provenance): 11 programming languages, an
agent/tool-calling pool, general web/wiki prose, a multilingual (DE/FR/IT)
pool, and a math pool. The corpus mirrors the deployment profile —
code-heavy with a 2×128k long-document tail — so the deepest context
regimes (128k/160k) are actually calibrated, not just short sequences.
Table with columns: Pool, Budget (tok), Used (tok), Seqs, Length range| Pool | Budget (tok) | Used (tok) | Seqs | Length range |
|---|
| code_typescript | 185,000 | 185,815 | 519 | 128–4,096 |
| code_python | 150,000 | 151,189 | 125 | 256–4,096 |
| code_javascript | 115,000 | 116,018 | 160 | 128–4,096 |
The 12 long tiers (four at 32k, four at 16k, two at 64k, two at 128k) add
754 documents reaching up to 132,968 tokens. During quantization GPTQModel
repacks the corpus into 40,960-token blocks, so no forward pass ever exceeds
40,960 tokens — this bounds the GDN fp32 value-stream tensor and removes the
OOM on the longest sequences structurally.
Run statistics
Table with columns: Stage, Wall time| Stage | Wall time |
|---|
| Model load (bf16, meta-init → GPU per layer) | 2 s |
| Quantize | 90.2 min |
| Save (6 shards) | 15 s |
Total wall time ~90 min on a single RTX 5090.
3. Quality verification
Three independent checks: the quantizer's own per-module loss, an absolute
perplexity, and a functional smoke suite run against the live server.
(a) Quantization loss (from quant_log.csv, 400 modules)
Table with columns: Metric, Value| Metric | Value |
|---|
| Modules quantized | 400 (48 GDN × 6 + 16 full-attn × 7) |
| Mean per-module loss | 1.108e-05 |
| Max per-module loss | 6.713e-05 |
| Worst module | L61 mlp.gate_proj |
| Worst cluster | mlp.gate_proj at layers 58–61 (~6.3–6.7e-05) |
| RTN fallbacks | 0 (fallback configured but never triggered) |
The per-module loss is the squared quantization error GPTQ minimizes per
linear layer. A max of 6.7e-05 across all 400 modules, with zero modules ever
falling back to the RTN (round-to-nearest) strategy, indicates a clean,
uniform fit — no layer required a degraded fallback to converge.
(b) Perplexity (absolute)
Table with columns: Metric, Value| Metric | Value |
|---|
| Perplexity | 2.729 |
| Tokens scored | n = 260 |
| Method | max_tokens=1 + prompt_logprobs=1 over a fixed, neutral English prose sample; PPL = exp(−mean logprob) over the input tokens |
| Runtime | live vLLM (this checkpoint) |
This is an absolute perplexity on a single 260-token sample — a sanity
check that the dequantized model assigns sensible probabilities to plain
prose, not a quality benchmark. The base model publishes no perplexity, so
there is nothing to diff against; the number is reported as-is.
(c) Functional smoke (3/3 pass, live server)
Table with columns: Probe, Result, Detail| Probe | Result | Detail |
|---|
| Coding generation | PASS | 2,273 chars in 11 s (enable_thinking: false) |
| Long-context needle | PASS | target located at 46,037-token context, returned ZEBRAX42 (2 s) |
| Tool / function calling | PASS | exact JSON: {"name":"get_weather","arguments":{"city":"Graz","unit":"celsius"}} (1 s) |
The needle probe exercises the long-context regime that the 2×128k calibration
tail was built to protect. The tool-call probe verifies the
qwen3_coder parser path end-to-end.
What is not verified here
- No base-vs-quant benchmark table. The base card publishes no
benchmarks or perplexity, and the 55.6 GB base cannot be co-resident with
this checkpoint on the 32 GB test GPU. If a head-to-head is needed, it
requires ≥80 GB of VRAM.
- MTP speculative decoding is preserved (15/15 tensors, bf16) but was
served without the speculative flag in the runs above; see §4.
4. Deployment
Verified serving configuration on vLLM 0.27.1 (Marlin GPTQ kernels, FP8 KV
cache, FlashInfer attention), 163,840-token context on an RTX 5090 32 GB:
vllm serve bernhardbrieger/Qwen3.8-27B-GPTQ-Int4 \
--tensor-parallel-size 1 \
--max-model-len 163840 \
--gpu-memory-utilization 0.90 \
--dtype bfloat16 \
--kv-cache-dtype fp8 \
--reasoning-parser qwen3 \
--tool-call-parser qwen3_coder \
--enable-auto-tool-choice \
--max-num-seqs 2 \
--max-num-batched-tokens 8192 \
--enable-prefix-caching \
--attention-backend FLASHINFER
- Weights: 19.54 GB resident. At 160k context with an FP8 KV cache the
server holds ~31.4 GB of the 32 GB GPU — it fits, with
max_num_seqs 2.
- Reasoning / tool parsing:
--reasoning-parser qwen3 +
--tool-call-parser qwen3_coder are what make structured tool-call output
work (see the smoke probe in §3c).
Optional: MTP speculative decoding
The MTP draft layer is present (15 tensors, bf16). To use it:
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
num_speculative_tokens: 2 is the Qwen-recommended value for a single MTP
layer. The MTP weights remain fully usable without the flag (they are
simply unused), so the checkpoint runs as a normal model on any setup —
speculative decoding is opt-in.
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model = AutoModelForCausalLM.from_pretrained(
"bernhardbrieger/Qwen3.8-27B-GPTQ-Int4",
torch_dtype=torch.bfloat16,
device_map="cuda",
)
tok = AutoTokenizer.from_pretrained("bernhardbrieger/Qwen3.8-27B-GPTQ-Int4")
One-shot measurement, vLLM 0.27.1, 2026-08-22, batch 1, on the exact
§4 flags above (RTX 5090 32 GB). Per context length: 1 warmup request +
3 timed runs, median reported, max_tokens=256, temperature 0, thinking
mode off. Prompts were drawn from the §2 calibration corpus such that no
prompt is a prefix of another — with --enable-prefix-caching active the
server-side hit rate over the whole run was 0.0 %, i.e. every number
below is full prefill. Prefill throughput = prompt tokens / TTFT; decode
throughput = completion tokens / (total time − TTFT). The KV-cache column
is derived from the model config, not measured: FP8 KV holds 32 KiB per
token (16 full-attention layers × K+V × 4 KV heads × 256 head-dim × 1
byte), so each entry equals its context length × 32 KiB.
Table with columns: Context, TTFT (s), Prefill (tok/s), Decode (tok/s), KV cache (GiB)| Context | TTFT (s) | Prefill (tok/s) | Decode (tok/s) | KV cache (GiB) |
|---|
| 8,192 | 2.118 | 3,873 | 80.2 | 0.25 |
| 16,384 | 4.418 | 3,711 | 79.8 | 0.50 |
| 32,768 | 9.434 | 3,475 | 77.8 | 1.00 |
|
Single-shot snapshot: absolute numbers will differ on other vLLM
releases (kernel and scheduling changes). Reproduce with
scripts/bench_serve.py against a server launched with the §4 flags.
5. Hardware requirements
Table with columns: Deployment, Minimum VRAM| Deployment | Minimum VRAM |
|---|
| Weights only (short context) | ~20 GB |
| 160k context, FP8 KV, 2 seqs (verified) | 32 GB (RTX 5090) |
The verified configuration uses the full 32 GB. The 19.54 GB weight
footprint is a fixed cost: a GPU below ~20 GB cannot load this checkpoint
at any context length. The remaining budget goes to the KV cache, which
scales with context length and concurrency — it determines how much
context a given card can serve.
6. Reproducibility appendix
Every number in §1–§3 is produced by one of the following. Paths are relative
to this repository.
Exact re-quantization needs the calibration corpus (the file referenced in
the code below), which is documented in provenance.json
but not redistributed with this checkpoint.
6.1 Quantization
The checkpoint is produced by GPTQModel over a 2,026-sequence corpus:
import json
from gptqmodel import GPTQConfig, GPTQModel
DYNAMIC = {
"-:.*\.input_layernorm$": {},
"-:.*\.post_attention_layernorm$": {},
"-:.*\.self_attn\.q_norm$": {},
"-:.*\.self_attn\.k_norm$": {},
"-:.*\.norm$": {},
"-:.*\.linear_attn\.conv1d$": {},
"-:.*\.linear_attn\.in_proj_a$": {},
"-:.*\.linear_attn\.in_proj_b$": {},
"-:.*\.embed_tokens$": {},
"-:(?:.*\.)?visual\..*": {},
"-:(?:.*\.)?mtp\..*": {},
}
model = GPTQModel.load(
"Qwen/Qwen3.8-27B",
GPTQConfig(bits=4, group_size=32, desc_act=False, sym=True,
lm_head=False, dynamic=DYNAMIC,
offload_to_disk=True, offload_to_disk_path="output/offload",
calibration_data_device="cpu"),
)
seqs = json.load(open("calibration_data.json"))["sequences"]
model.quantize(seqs, batch_size=1,
calibration_sort="desc",
calibration_concat_size=40960)
model.save("output/Qwen3.8-27B-GPTQ-Int4")
offload_to_disk streams finished layers to NVMe during the layer loop (the
default would otherwise stage them on the GPU). Per-module losses land in
quant_log.csv; the resolved config is written to quantize_config.json.
6.2 Perplexity
Against a running server on :8000, scoring the fixed 260-token sample with
input-token logprobs:
python scripts/measure_ppl_quant.py
# -> quant_ppl n=260 perplexity=2.729
The script POSTs the fixed sample with max_tokens=1, prompt_logprobs=1 and
computes exp(-mean logprob) over the prompt positions.
6.3 Functional smoke
python scripts/smoke_test_api.py
# -> 3 probes: coding / needle_40k / tool_call (results in smoke_test_api.jsonl)
7. License & provenance
Model weights. Apache-2.0. The base
Qwen/Qwen3.8-27B is Apache-2.0 and
this derived checkpoint is released under the same license.
Calibration data. The corpus is drawn entirely from public, non-gated
datasets. Every source, its license, and (for code) its pinned commit are
listed below; the full manifest is provenance.json in
this repository. The raw calibration sequences are not redistributed with
this checkpoint.
Table with columns: Source, License, Used in| Source | License | Used in |
|---|
| 28 code repositories | Apache-2.0 (13) · MIT (13) · BSD-2-Clause (1) · BSD-3-Clause (1) | code pools (11 languages) |
| codeparrot/codeparrot-clean | per-file upstream license (no dataset-level license declared) | Python pool |
| wikimedia/wikipedia (EN/DE/FR/IT) | CC BY-SA 3.0 + GFDL | wiki prose, long-document tail |
| openai/gsm8k | MIT |
Each code repository is pinned to a specific commit in
provenance.json (repo + commit SHA + license), so the
code portion of the corpus is reconstructable byte-for-byte.
Data attribution. Wikipedia text © Wikipedia contributors, released under
CC BY-SA 3.0 and the GFDL. FineWeb-Edu and the Tülu-3 SFT mixture are used
under ODC-BY. Attribution is provided here in lieu of per-file notices.
Base model references.