pearsonkyle

Qwen3.8-27B-GPTQ-W4A16

README

License: apache-2.0

🚀 Quick start

bash
vllm serve pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 \
    --max-model-len 32768 \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml

--tool-call-parser qwen3_xml is required for tool calling. Qwen3.8 emits XML tool calls (<tool_call><function=NAME>), not JSON. Without it tool_calls is always empty — which looks like a broken quant but is a serving flag.

Benchmarks

All on vLLM 0.27.1 (RTX PRO 6000 Blackwell) vs a bf16 reference served identically. That control is the only valid comparison — the same reference scores 0.563 here vs 0.494 on llama.cpp, so these do not compare across stacks.

Agentic — 25 held-out sessions / 174 turns, greedy, thinking off

Table with columns: bf16 (ref), W4A16
	bf16 (ref)	W4A16
Tool-selection accuracy	0.563 (98/174)	0.563 (98/174)
Parameter accuracy	0.345	0.350
Schema-valid rate	0.948	0.948 (165/174)
Malformed tool calls	0	0

SWE-rebench (dask__dask-11393, end-to-end through the OpenAI Agents SDK, thinking off): ✅ resolved · 1/1 FAIL_TO_PASS · 34/34 PASS_TO_PASS · 5 steps, 622 output tokens · 0 malformed commands.

Long-context retrieval: 3/3 rare codes recovered exactly at ~29.7k tokens (PINEAPPLE-7742, ZEPHYR-3391-QX, MARMALADE-8156).

Vision: 3/3 on colour + shape + position for a synthetic three-shape image.

Static fidelity: median KLD vs bf16 across six held-out distributions ranges 0.0021 (agentic) to 0.0205 (general English) at eval-ctx 8192 — lowest exactly where the model was calibrated.

Table with columns: eval, what it is, median KLD, top-1 agree, top-5 agree, ppl (bf16 → W4A16)
eval	what it is	median KLD	top-1 agree	top-5 agree	ppl (bf16 → W4A16)
`external`	code + math + tools, disjoint from calibration — the headline	0.0141	87.8%	97.4%	15.186 → 16.094
`general`	broad English (`combined_en_tiny`)	0.0205	86.1%

KLD is 3–10× lower on agentic/tool text than on general English — calibration spending fidelity where the model is used. Worst case for general chat: general, 0.0205.

Perplexity improves on three of six — that is noise, not a win. Trust KLD and top-1.

Reasoning level

Set per-request, no restart:

python
extra_body={"chat_template_kwargs": {"reasoning_effort": "medium"}}

Measured over 174 tool-call turns at max_tokens=512, tool-selection accuracy ranges 0.437 (xhigh) to 0.563 (off) — a 22-turn spread. Only the endpoints are reliable: replaying the ladder moved levels by up to 6 turns, and high/medium/low do not order consistently.

Table with columns: Reasoning level, What it injects, Tool-sel acc, Turns, Param acc, Schema
Reasoning level	What it injects	Tool-sel acc	Turns	Param acc	Schema
`xhigh` (default)	think carefully + validate assumptions + consider alternatives	0.437	76/174	0.226	0.862
`high`	think carefully + validate assumptions	0.483	84/174	0.258

Read the endpoints, not the ordering. Run twice on the same weights, levels moved by up to 6 turns — so only the ends are safe: off is clearly best, xhigh clearly worst (22-turn spread), and high/medium/low are one blur that does not order consistently.

Tool-calling on a tight budget → enable_thinking: false. Reasoning competes with the answer for the same max_tokens; raise the budget before the reasoning level.

At 512 tokens xhigh is truncated before it can emit a <tool_call>, which the scorer records as "no tool call" — a serving artifact, not model quality. medium injects no instruction (native reasoning).

The stock template raises on reasoning_effort: "high" — the OpenAI-standard value — so a normal OpenAI client gets HTTP 400. The bundled chat_template_safe_v2.jinja fixes that plus three rendering bugs, and is byte-identical on 382/382 real holdout prefixes, so adopting it cannot change quality:

bash
vllm serve ... --chat-template chat_template_safe_v2.jinja

MTP speculative decoding

The trained MTP draft head ships inside the checkpoint at bf16 — no second file to download.

bash
vllm serve pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 \
    --max-model-len 32768 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":2}'

Table with columns: draft-n, decode tok/s, vs baseline, acceptance
draft-n	decode tok/s	vs baseline	acceptance
off (baseline)	75.0	1.00×	—
1	113.6	1.51×	82.1%
2	128.3	1.71×	71.2%

Best setting: num_speculative_tokens: 2 — 1.71× decode at 71.2% acceptance. Qwen3.8 has one nextn layer, so a deeper draft re-runs that head on its own guess and per-token acceptance drops — but it still nets more accepted tokens per step, which is what throughput follows. Optimise tokens-gained-per-step, not acceptance rate.

GPTQ W4A16 via llm-compressor 0.13.0 → compressed-tensors 0.18.0: int4, group 128, symmetric, static act-order. Sequential pipeline over all 64 layers (48 linear-attention + 16 full-attention), 496 modules quantized.

Kept bf16 on purpose: lm_head, embed_tokens, vision tower, MTP head. The 248,320-token vocab with untied embeddings makes the two vocab tensors 4.74 GiB, 26% of the download — which is why a "4-bit" 27B lands at 5.2 bpw overall (trunk alone: 4.05) and is bigger than the 14.5 GiB IQ4_XS GGUF. A quantized head over a 248k vocab is the classic rare-token failure mode; the needle test is the check that this worked.

Calibration: 4,255,761 tokens / 3,436 windows at ctx 32,768 — 63% real agentic sessions (CLI logs + SWE trajectories), 6,786 tool calls across 76 schemas, plus reasoning turns, broad-instruct and red-team refusals. GPTQ drew 128 × 32,768-token sequences by deterministic whole-corpus stride.

Built with Quant-Tuner; logs mined with LogMiner.

Limits

Images tested, video not. Multi-image, high-res grounding and vision at long context are untested.
Vision tower and MTP head are bf16, not quantized (+1.65 GiB) — which is why they work.
SWE-rebench is one instance, not a pass rate.
Agentic deltas are noise-bound. Replaying the ladder twice on the same weights moved levels by up to 6 turns (3.4pp) — vLLM is nondeterministic at temperature=0. Read the endpoints, not the third decimal.

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

pearsonkyle

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality