halt95

Qwen3.8-27B-W4A16-Merlin

README

License: apache-2.0

The checkpoint

Table with columns: Component, Format
Component	Format
Gated attention (16 layers), Gated DeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj` (48 layers) and feed-forward (all 64)	AWQ INT4, asymmetric, group 128 (philbert's checkpoint)
`lm_head`	INT4 GPTQ, asymmetric, group 128
MTP draft head (projections, attention, feed-forward)	INT4, asymmetric, group 128, GPTQ-calibrated on drafter inputs
Gated DeltaNet `in_proj_a` / `in_proj_b`, conv and norms; vision tower; embeddings; norms	BF16, unchanged
KV cache	FP8 E4M3 with per-layer static scales for the 16 attention layers and the MTP layer (`kv_scales.safetensors`)

Files in this repository

Table with columns: Files, What they hold, Size
Files	What they hold	Size
`model.safetensors`	The language model, vision tower and embeddings	15.7 GiB
`model-mtp.safetensors`	The MTP draft head, INT4	210 MiB
`kv_scales.safetensors`	Calibrated static scales for the FP8 KV cache	3.5 KiB
`model.safetensors.index.json`	Which file holds each tensor	206 KiB
`chat_template.jinja`	Qwen's own chat template, unchanged (byte-identical to the upstream file). Merlin serves Froggeric's fixed Qwen3.8 template (v22.5) by default; see Use it	8.7 KiB
`config.json`, `generation_config.json`	Model, quantisation and sampling settings	21.4 KiB
`tokenizer.json`, `tokenizer_config.json`, processor configs	`tokenizer.json` byte for byte from Qwen3.8-27B; `tokenizer_config.json` re-serialised by transformers; image / video processor settings	12.2 MiB

About 16 GiB in all (the 11 files of the engine's models/qwen38-27b/checkpoint.sha256). Also in the repository: LICENSE (Apache-2.0) and NOTICE.

Speed

Table with columns: Prompt, Decode, 1× 3090, Decode, 2× 3090, Decode, 4× 3090
Prompt	Decode, 1× 3090	Decode, 2× 3090	Decode, 4× 3090
4K	125 tok/s	210 tok/s	251 tok/s
32K	120 tok/s	199 tok/s	259 tok/s
131K	95 tok/s	182 tok/s	243 tok/s
262K	71 tok/s	143 tok/s	238 tok/s

One stream, thinking on (effort medium), ≤ 256 tokens out (128 at 262K), median of 3; 4,046 / 32,566 / 130,386 / 261,888 prompt tokens. Thinking off at 262K: 50 / 109 / 149 tok/s on one / two / four cards. On four and two cards that is lower than in 3.0.0 (158 and 111 tok/s, −5.7 % and −2.1 %), while the 259K thinking-off ladder cells read +5.4 % and +4.8 %: the time per decode step is the same or shorter (four cards 20.00 ms against 20.78 ms), and the gap is MTP draft acceptance over a 128-token sample (2.84 against 3.12 tokens per step on four cards). The thinking-on +11.0 % at 262K on four cards is mostly the same effect in the other direction: draft acceptance over the same 128-token sample (4.41 against 4.13 tokens per step). The engine wheel is the same file.

Table with columns: Prompt, Time to first token, 1× 3090, 2× 3090, 4× 3090
Prompt	Time to first token, 1× 3090	2× 3090	4× 3090
4K	3.32 s	1.73 s	1.08 s
32K	29.1 s	15.0 s	8.86 s
131K	163.6 s	84.7 s	46.8 s
262K	452.4 s	234.9 s	124.7 s

Time to first token (TTFT): the median of 3 cold sends below 262K (the thinking-on ladder cells) and the cold send 1 at 262K. Four cards at 262K: 124.7 s, against 164.1 s in 3.0.0. Prefill speed at 10K, 100K and 262K is in the next table.

Table with columns: Cold prefill, one send, 1× 3090, 2× 3090, 4× 3090
Cold prefill, one send	1× 3090	2× 3090	4× 3090
10,157 tokens	1,200 tok/s · 8.47 s	2,301 tok/s · 4.41 s	3,866 tok/s · 2.63 s
101,845 tokens	871 tok/s · 116.9 s	1,672 tok/s · 60.9 s	2,995 tok/s · 34.0 s
261,888 tokens	579 tok/s · 452.4 s	1,115 tok/s · 234.9 s	2,101 tok/s · 124.7 s

Prefill and time to first token, one cold request per prompt, 8 tokens out; the 261,888-token row is the exact-262K prompt's cold thinking-on send 1 (128 tokens out; the 2026-11-01 run).

Table with columns: Agents at once, 1× 3090, 2× 3090, 4× 3090
Agents at once	1× 3090	2× 3090	4× 3090
2 (3.0.0 run)	164 tok/s	281 tok/s	361 tok/s
4 (3.0.0 run)	two at most	373 tok/s	549 tok/s
8, thinking off	two at most	615 tok/s	854 tok/s
8, thinking on (effort medium)	two at most	705 tok/s	954 tok/s

N 16K conversations, then N simultaneous 512-token follow-ups, total tok/s while all N decode; thinking off unless stated. One card runs two sequences at most. The 8-agent rows are 3.0.1 (2026-11-01 to 2026-11-02); with thinking on a 512-token follow-up may end inside the reasoning, and reasoning tokens count, so those totals measure decoded tokens, not finished answers. Overlap windows: four cards 4.30 s thinking off and 3.55 s on, two cards 6.08 s and 4.92 s; read the totals from windows under 5 s loosely. The 2- and 4-agent rows were not re-run: 3.0.0 runs (one and two cards: a pre-release engine build, 2026-10-17; four cards: the release wheel, 2026-10-20), windows 2.6 to 5.7 s.

Table with columns: Sessions held at once, Needles found, Re-sent from cache, lowest
Sessions held at once	Needles found	Re-sent from cache, lowest
1 × 262K or 2 × 131K (one card)	1 / 1 · 2 / 2	99.99 % · 99.98 %
3 × 262K or 6 × 131K (two cards)	3 / 3 · 6 / 6	99.99 % · 99.98 %
7 × 262K or 14 × 131K (four cards)	7 / 7 · 14 / 14	99.99 % · 99.98 %

N needle prompts (thinking off, greedy) sent at once, then re-sent one by one: every re-send served from the prefix cache means the pool held all N. 0 preemptions. 3.0.0 run, not re-run for 3.0.1: the KV pool and the sessions it holds are unchanged.

Table with columns: Measure, 1× 3090, 2× 3090, 4× 3090
Measure	1× 3090	2× 3090	4× 3090
Decode, 4K prompt	124 → 125 (+1.1 %)	207 → 210 (+1.9 %)	250 → 251 (+0.1 %)
Decode, 131K prompt	95 → 95 (+0.4 %)	173 → 182 (+5.1 %)	235 → 243 (+3.1 %)
Decode, 262K prompt	73 → 71 (−2.1 %)	142 → 143 (+0.7 %)	215 → 238 (+11.0 %)
Decode, 262K, thinking off	49 → 50 (+2.4 %)	111 → 109 (−2.1 %)	158 → 149 (−5.7 %)
Cold prefill, 10,157 tokens	1,021 → 1,200 (+17.5 %)	1,823 → 2,301 (+26.2 %)	3,009 → 3,866 (+28.5 %)
Cold prefill, 101,845 tokens	708 → 871 (+23.0 %)	1,297 → 1,672 (+28.9 %)	2,295 → 2,995 (+30.5 %)
Cold prefill, 261,888 tokens	456 → 579 (+27.0 %)	858 → 1,115 (+29.9 %)	1,596 → 2,101 (+31.6 %)
8 agents, thinking off	n/a	517 → 615 (+18.9 %)	752 → 854 (+13.6 %)

3.0.0 → 3.0.1 (change), tok/s; decode is one stream with thinking on at effort medium on both releases unless stated, prefill is one cold send. Cold time to first token at 262K: 574.1 → 452.4 s on one card, 305.3 → 234.9 s on two, 164.1 → 124.7 s on four. Where the settings differ: the 3.0.1 runs used the optional GPU-unlock host upgrade and the 3.0.0 runs did not (in a stock-versus-unlocked A/B on two cards at 262K the unlock alone made prefill about 29 % faster, with decode unchanged and identical output); the engine wheel is the same file; the 3.0.0 two-card 8-agent total came from a pre-release engine build (2026-10-17), the four-card one from the release wheel. Both four-card agent totals come from windows under 5 s. The 3.0.0 card has the full 3.0.0 tables.

Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02; the 3.0.1 runs also with the optional Zagitta's gpu-unlock host upgrade.
Engine: Merlin 3.0.1 (the engine wheel is unchanged from 3.0.0), served by merlin serve with the shipped defaults, one fresh boot per layout. One card: TP1; two cards: TP2 on GPUs 0 and 2; four: TP4. Runs 2026-11-01 to 2026-11-02, on release candidates just before the final commit, not on the final commit itself.
Layout: MTP K=4, FP8 KV, RecoverSSM on every layout, host-resident token embeddings, thinking effort medium (the server default).
Decode: T=0, a needle prompt at depth, every send a cold prefill except sends 2 and 3 at 262K, which reuse the prefix before the needle; one discarded warm-up, then 3 sends per cell, median. tok/s = tokens after the first streamed event / time from first to last event.
Streams are client-timed. Full tables, thinking-off rows and ITL: the bench card.

Quality

This checkpoint, measured on the Merlin engine:

Table with columns: Check, Result, Setting
Check	Result	Setting
GSM8K-200	195 / 194 / 197 of 200 on 1 / 2 / 4 cards	5-shot, thinking off
GSM8K-200, thinking on	198 / 200 on four cards	5-shot, effort medium, none cut at the 8,192-token cap
Needles at 131K	5 / 5 on each layout	single-occurrence needles, an image in the prompt, thinking off, greedy, cold cache
Needles at 262K	12 / 12 on each layout	as above
Needle at exactly 261,888 tokens	3 / 3 on each layout	thinking off
Image, 4096 × 2048 (8,388,608 px)	1 / 1 · 2 / 2 at once · 1 / 1 with four in one prompt, on 2 and 4 cards	thinking off; names both colours of a two-colour image (with four, of the last one)

Thinking-on GSM8K-200: Merlin 3.0.1, four cards, 2026-11-01 to 2026-11-02, the same 200 problems and scorer as the thinking-off row, temperature 0. Every other row is a 3.0.0 run, not re-run for 3.0.1. Four cards and images: the Merlin 3.0.0 release wheel, 2026-10-20. One and two cards' GSM8K and 131K / 262K needle panel: a pre-release engine build, 2026-10-17; the exact-261,888-token needle on every layout: the release wheel, 2026-10-20. These are checks of this checkpoint, not a matched BF16 comparison: no BF16 run of the same checks was made. The FP8 KV scales were calibrated on traffic that included the first 50 GSM8K test questions (activation maxima only); the lm_head and MTP head were GPTQ-calibrated on public benchmark prompts (ARC, MMLU-Pro, GPQA, GSM8K train split, HumanEval), synthetic tasks and a small hand-written set. No calibration data is distributed.

The base model, as reported by Qwen for the BF16 weights:

Table with columns: Area, Benchmark, Qwen3.8-27B (BF16)
Area	Benchmark	Qwen3.8-27B (BF16)
Coding	Terminal Bench 2.1 (Terminus)	73.0
Coding	SWE-bench Pro	61.7
Coding	LiveCodeBench v6	90.3
Agents	CoWorkBench	70.7
Reasoning and instructions	GPQA Diamond	89.2
Reasoning and instructions	HLE	30.8
Reasoning and instructions	IFBench	79.5
Vision	OSWorld-Verified	84.3
Vision	AndroidWorld	81.9
Vision	RealWorldQA	85.9

A selection from the upstream card, which has the full tables and comparisons with other models. These were not re-run on this 4-bit build.

What it fits on

Table with columns: Measured on, Context per request, KV pool, Held at once, Images
Measured on	Context per request	KV pool	Held at once	Images
1× RTX 3090, 24 GB (🟢 Tested)	262,144 tokens	271,506 tokens	1 × 262K or 2 × 131K	1 per prompt, 4 MP
2× RTX 3090, 24 GB (🟢 Tested)	262,144 tokens	826,689 tokens	3 × 262K or 6 × 131K	8 MP
4× RTX 3090, 24 GB (🟢 Tested)	262,144 tokens	2,012,105 tokens	7 × 262K or 14 × 131K	8 MP

Measured on the reference host: the KV pools as in the 2026-11-02 bench card; the sessions held at once and the image figures are 3.0.0 runs (2026-10-20 bench card), not re-run for 3.0.1. KV cache FP8, the pool all in VRAM. Two and four cards need GPU peer-to-peer on RTX 3090s for these rows (hosts without it: the no-P2P profiles below). A 4096 × 2048 image (8,388,608 px) left at least 886 MiB free on two cards and 862 MiB on four. One card takes one 4 MP image per prompt: with the single-card memory settings, one and two requests at once and after a 262K needle left at least 558 MiB free (2026-10-21). The original one-card run failed at 8 MP; with those settings 8 MP still stays below the 524 MiB headroom floor (478 MiB on the reference host).

Other GPUs. The engine also ships profiles for these; the Status column says how far each has been run:

Table with columns: GPUs, Arch, Profile, Context fit, Status
GPUs	Arch	Profile	Context fit	Status
2 / 4× RTX 3090 without P2P	Ampere, sm_86	`27b-tp2-3090-nop2p`, `27b-tp4-3090-nop2p`	~3 / ~7 × 262K (the 3090 pins)	⚫ Untested
1 / 2 / 4× RTX 3090 Ti, RTX A5000 (24 GB)	Ampere, sm_86	`27b-tp1-3090`, `-tp2-3090`, `-tp4-3090`	~1 / ~3 / ~7 × 262K (as the 3090)	⚫ Untested
1 / 2 / 4× RTX 4090 (24 GB)	Ada, sm_89	`27b-tp1-sm89-24g`, `-tp2-`, `-tp4-`	~1 / ~3 / 7 × 262K (four cards: 7 × 262K held on a rented host with Qwen3.8-27B; one card: the full run ran out of memory at chunk 1024; at the shipped chunk 512 only the 131K and 262K needles have run)	🟡 Testing on 2×, 4×; ⚫ otherwise
1 / 2 / 4× 32 / 48 GB Ada: RTX 4080 Super (modified), RTX 4090 48 GB (modified), RTX 6000 Ada, L40S	Ada, sm_89	`27b-tpN-sm89-32g`, `-sm89-48g`	~1 / ~3 / ~7 × 262K (the 24 GB pins)	⚫ Untested
1 / 2 / 4× 48 GB: RTX A6000, A40	Ampere, sm_86	`27b-tpN-sm86-48g`	~3 / ~8 / ~18 × 262K	⚫ Untested
1 / 2 / 4× A30 (24 GB)	Ampere, sm_80	`27b-tpN-sm80-24g`	~1 / ~3 / ~7 × 262K	⚫ Untested
1 / 2 / 4× CMP 170HX 32 GB	Ampere, sm_80	`27b-tpN-sm80-32g`	~1 / ~4 / ~11 × 262K	⚫ Untested
1 / 2 / 4× A100 40 GB	Ampere, sm_80	`27b-tpN-sm80-40g`	~2 / ~6 / ~14 × 262K	⚫ Untested
1 / 2 / 4× CMP 170HX 64 GB	Ampere, sm_80	`27b-tpN-sm80-64g`	~5 / ~11 / ~25 × 262K (needles exact on a rented host)	🟡 Testing
1 / 2 / 4× A100 80 GB	Ampere, sm_80	`27b-tpN-sm80-80g`	~7 / ~15 / ~32 × 262K	⚫ Untested

merlin serve and ./setup.sh pick the profile from the GPUs they find; without P2P the default is the largest layout, and merlin serve prints the --tp 1 --instances N command, which gives more total throughput for many users. A ~ is the engine's estimate, not a measurement. Every profile uses an FP8 KV cache, on sm_80 cards too (FP8 KV ran on the 4× CMP 170HX 64 GB rental on one, two and four cards, needles exact, GSM8K-50 48/50). 🟢 measured by us · ⚫ a profile ships, not yet qualified by us (expected to work; a rented-host run is noted in the row where one was made). Every profile: the engine's hardware matrix. 🟡 Testing: full-model runs passed on cloud-hosted cards; not yet qualified against the reference host.

Table with columns: Need, Details
Need	Details
Host RAM	About 2.4 GiB pinned for the host-resident token embeddings (1.18 GiB per card on two cards, 0.59 on four), on top of normal use; one card also keeps the vision-tower weights in host RAM (about 1 GiB)
KV RAM cache (optional)	Off by default. `--kv-ram-cache SIZE` pins SIZE of host RAM on top of the RAM above, at least 9 GiB on one card, 10 on two and 12 on four; see Use it
Disk	About 16 GiB for the checkpoint, plus the engine environment and caches
Engine host	Linux x86-64 (glibc 2.38 or newer), NVIDIA driver R580 or later (CUDA 13), Python 3.13, GCC/G++ 13 or newer, `/dev/shm` of at least 1 GB; GPUs headless with no other CUDA process. `merlin doctor` checks the host.

Use it

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine./setup.sh --model 27b

One command does the rest: after a small uv and Python bootstrap it checks the GPUs, driver, RAM and disk, shows the engine, package and checkpoint downloads with their sizes before fetching them, installs the prebuilt engine, downloads and verifies this checkpoint, and starts an OpenAI-compatible server on 127.0.0.1:8100. Enter takes the recommended answer at each question; --yes takes them all, --dry-run shows the plan. Run ./setup.sh again to start the server, and git pull && ./setup.sh to update the engine without downloading the model again. Requests use the model name qwen38-27b; the setup asks how many cards to use and recommends the largest tested layout (on a multi-card host without P2P it recommends --tp 1 --instances N).

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-enginecurl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whlMERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" \  bash release/install-env.sh ./venv venv/bin/hf download halt95/Qwen3.8-27B-W4A16-Merlin --revision 7a0af39f63dd4fb0733c23d3ec126762074a503f \  --local-dir ~/models/27b(cd ~/models/27b && sha256sum -c "$OLDPWD/models/qwen38-27b/checkpoint.sha256")venv/bin/python merlin_cli.py serve ~/models/27b

MERLIN_WHEEL installs the prebuilt engine (sm_80, sm_86, sm_89; the same wheel file as in Merlin 3.0.0), checked against release/merlin-wheel.sha256; without it the installer compiles the engine instead, about 1.5 hours on 8 cores. merlin serve recognises the checkpoint, picks the largest tested layout for the GPUs it finds (or, with none, the largest untested one that fits, labelled as such) and serves on 127.0.0.1:8100. --tp 1|2|4 picks a layout. Add --dry-run to print the resolved command without starting anything. The first start compiles kernels.

Recommended settings (Qwen's, for Qwen3.8-27B):

Table with columns: Mode, temperature, top_p, top_k, presence_penalty
Mode	temperature	top_p	top_k	presence_penalty
Thinking (default)	1.0	0.95	20	0.0
Non-thinking	0.7	0.80	20	1.5

Qwen's recommendation for this model, with min_p=0.0 and repetition_penalty=1.0 in both modes. A presence_penalty between 0 and 2 reduces endless repetition; higher values can mix languages.

Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
Reasoning effort: Merlin serves medium by default; pick another per request with "chat_template_kwargs": {"reasoning_effort": "low"} (low / medium / xhigh).
KV RAM cache (experimental, off by default): --kv-ram-cache SIZE on merlin serve or ./setup.sh keeps the prefix KV that the GPU evicts in pinned host RAM and restores it, with the same output, instead of recomputing it. It adds no context or sessions; SIZE comes on top of the host RAM above (at least 9g on one card, 10g on two, 12g on four). On four cards --swarm raises the session cap from 24 to 32 and sets a 48g cache. Under heavy concurrent prefill a session may recompute instead of restoring (output identical).
--spec dflash2 is the Flash-Next family's drafter option; it is refused on this checkpoint, which runs its own MTP head.
Chat template: Merlin serves Froggeric's fixed Qwen3.8 template (v22.5) by default, not the file in this repository. It accepts system messages after the first turn, so coding agents' per-turn notes stay in place and the prefix cache keeps hitting. CHAT_TEMPLATE=<path> serves another template; CHAT_TEMPLATE= (empty) serves this repository's own.
With Qwen's own template (CHAT_TEMPLATE=), earlier turns' reasoning is kept by default and "chat_template_kwargs": {"preserve_thinking": false} keeps only the latest.
Images: up to 64 per prompt at 8 MP each on two or four cards; one card takes one image per prompt at 4 MP. The cap is a served default: a client that raises it per request goes past the measured headroom.
Tool calls use the standard tools field, parsed with the Qwen3 coder tool parser.
To reach it from other machines, answer network at setup's Access question (it generates an API key), or start merlin serve with HOST=0.0.0.0 and VLLM_API_KEY set; put a TLS reverse proxy in front before opening it beyond your own network.

The base model

Qwen3.8-27B is Qwen's dense model on the Qwen3.5 architecture, with vision. The architecture, in short:

Hybrid attention. Three Gated DeltaNet (linear-attention) layers for every gated-attention layer: 48 and 16 of the 64 layers.
Dense. All 27B parameters work on every token; no experts.
Multi-token prediction. One MTP layer, trained over several steps; Merlin uses it for speculative decoding (4 draft tokens).
Thinking by default, with a non-thinking mode per request and reasoning_effort levels; Qwen's template keeps earlier turns' reasoning by default (preserved thinking).

Table with columns: Property, Qwen3.8-27B
Property	Qwen3.8-27B
Type	Causal language model with a vision encoder (images and video)
Parameters	27B
Layers	64: 16 × (3 × Gated DeltaNet → FFN, then 1 × gated attention → FFN)
Attention	24 query / 4 KV heads, head dim 256 · Gated DeltaNet: 48 V / 16 QK heads, head dim 128
Feed-forward	Width 17,408
Context	262,144 tokens native (Qwen: extensible to 1M; this build serves 262,144)

Architecture as reported by Qwen on the upstream card.

Other builds

Table with columns: Checkpoint, Base, GPUs, Decode at 262K
Checkpoint	Base	GPUs	Decode at 262K
Qwen3.8-27B (this card)	Qwen3.8-27B, dense	1×, 2× or 4× 3090	71 / 143 / 238 tok/s
Flash-Next	Qwen3.8-Flash-Next, 125B MoE	4× 3090	197 tok/s
Swift1.5	Flash-Next fine-tune	4× 3090	186 tok/s
Swift1.5-Huihui	Swift1.5, abliterated	4× 3090	196 tok/s
ThinkingCap	Qwen3.8-27B fine-tune	2× or 4× 3090	160 / 238 tok/s
Nemotron 3.5 Super VL	Nemotron 3.5 Super VL, 120B-A12B MoE	4× 3090	93 tok/s
Ornith 1.5	Qwen3.5-122B-A10B fine-tune (ornith-ai), 122B MoE	4× 3090	105 tok/s

One stream, thinking on, 261,888 prompt tokens + 128 out, median of 3; Merlin 3.0.1, 2026-11-02 bench cards. Effort as each bench card ran it: medium on every Qwen3.8 checkpoint (Flash-Next, Swift1.5, Swift1.5-Huihui, ThinkingCap and Qwen3.8-27B), the server default. Nemotron and Ornith 1.5 ran at full thinking: neither template reads reasoning_effort.

Good to know

Every figure we measured for this checkpoint comes from 4× RTX 3090 at 220 W, PCIe Gen4 with P2P, used as one, two or four cards, except the rental rows under Other GPUs; the 3.0.1 figures were also measured with the optional GPU-unlock host upgrade, the 3.0.0 ones were not; the Other builds table names each build's card count.
The same engine profiles serve ThinkingCap, BottleCap AI's fine-tune of this model, which reasons in fewer tokens (under a different licence).
Coming from the qwen38-27b-3090s engine (v1.x)? It is replaced by the Merlin engine; this checkpoint's files are unchanged.

Changes

Table with columns: Date, Change to this repository
Date	Change to this repository
2026-11-02	Card updated for Merlin 3.0.1 (speed figures, install tag, KV RAM cache note). The weights, config and tokenizer are unchanged
2026-10-12	`tokenizer.json` replaced with the upstream Qwen3.8-27B file: combining-mark scripts (Hindi, Thai, Arabic and others) now tokenize as upstream with transformers 5.18.0 or newer (which Merlin pins; 5.17.0 and earlier ignore this file); English, code and CJK are unchanged
2026-09-27	Weights published; unchanged since

Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.

Credits and licence

Qwen (model; Apache-2.0, see LICENSE), philbert (the AWQ INT4 body), froggeric (the fixed Qwen3.8 chat template the engine serves), the vLLM project, FlashInfer, compressed-tensors, aikitoria (peer-to-peer on consumer Ampere), Neiko2002, and CoolShades (the idle-power report behind the v2.5.2 engine fix). Upstream fixes are credited in the engine repository.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

halt95

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

The checkpoint

Table with columns: Component, Format
Component	Format
Gated attention (16 layers), Gated DeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj` (48 layers) and feed-forward (all 64)	AWQ INT4, asymmetric, group 128 (philbert's checkpoint)
`lm_head`	INT4 GPTQ, asymmetric, group 128
MTP draft head (projections, attention, feed-forward)	INT4, asymmetric, group 128, GPTQ-calibrated on drafter inputs
Gated DeltaNet `in_proj_a` / `in_proj_b`, conv and norms; vision tower; embeddings; norms	BF16, unchanged
KV cache	FP8 E4M3 with per-layer static scales for the 16 attention layers and the MTP layer (`kv_scales.safetensors`)

Files in this repository

Table with columns: Files, What they hold, Size
Files	What they hold	Size
`model.safetensors`	The language model, vision tower and embeddings	15.7 GiB
`model-mtp.safetensors`	The MTP draft head, INT4	210 MiB
`kv_scales.safetensors`	Calibrated static scales for the FP8 KV cache	3.5 KiB
`model.safetensors.index.json`	Which file holds each tensor	206 KiB
`chat_template.jinja`	Qwen's own chat template, unchanged (byte-identical to the upstream file). Merlin serves Froggeric's fixed Qwen3.8 template (v22.5) by default; see Use it	8.7 KiB
`config.json`, `generation_config.json`	Model, quantisation and sampling settings	21.4 KiB
`tokenizer.json`, `tokenizer_config.json`, processor configs	`tokenizer.json` byte for byte from Qwen3.8-27B; `tokenizer_config.json` re-serialised by transformers; image / video processor settings	12.2 MiB

About 16 GiB in all (the 11 files of the engine's models/qwen38-27b/checkpoint.sha256). Also in the repository: LICENSE (Apache-2.0) and NOTICE.

Speed

Table with columns: Prompt, Decode, 1× 3090, Decode, 2× 3090, Decode, 4× 3090
Prompt	Decode, 1× 3090	Decode, 2× 3090	Decode, 4× 3090
4K	125 tok/s	210 tok/s	251 tok/s
32K	120 tok/s	199 tok/s	259 tok/s
131K	95 tok/s	182 tok/s	243 tok/s
262K	71 tok/s	143 tok/s	238 tok/s

Table with columns: Prompt, Time to first token, 1× 3090, 2× 3090, 4× 3090
Prompt	Time to first token, 1× 3090	2× 3090	4× 3090
4K	3.32 s	1.73 s	1.08 s
32K	29.1 s	15.0 s	8.86 s
131K	163.6 s	84.7 s	46.8 s
262K	452.4 s	234.9 s	124.7 s

Table with columns: Cold prefill, one send, 1× 3090, 2× 3090, 4× 3090
Cold prefill, one send	1× 3090	2× 3090	4× 3090
10,157 tokens	1,200 tok/s · 8.47 s	2,301 tok/s · 4.41 s	3,866 tok/s · 2.63 s
101,845 tokens	871 tok/s · 116.9 s	1,672 tok/s · 60.9 s	2,995 tok/s · 34.0 s
261,888 tokens	579 tok/s · 452.4 s	1,115 tok/s · 234.9 s	2,101 tok/s · 124.7 s

Prefill and time to first token, one cold request per prompt, 8 tokens out; the 261,888-token row is the exact-262K prompt's cold thinking-on send 1 (128 tokens out; the 2026-11-01 run).

Table with columns: Agents at once, 1× 3090, 2× 3090, 4× 3090
Agents at once	1× 3090	2× 3090	4× 3090
2 (3.0.0 run)	164 tok/s	281 tok/s	361 tok/s
4 (3.0.0 run)	two at most	373 tok/s	549 tok/s
8, thinking off	two at most	615 tok/s	854 tok/s
8, thinking on (effort medium)	two at most	705 tok/s	954 tok/s

Table with columns: Sessions held at once, Needles found, Re-sent from cache, lowest
Sessions held at once	Needles found	Re-sent from cache, lowest
1 × 262K or 2 × 131K (one card)	1 / 1 · 2 / 2	99.99 % · 99.98 %
3 × 262K or 6 × 131K (two cards)	3 / 3 · 6 / 6	99.99 % · 99.98 %
7 × 262K or 14 × 131K (four cards)	7 / 7 · 14 / 14	99.99 % · 99.98 %

Table with columns: Measure, 1× 3090, 2× 3090, 4× 3090
Measure	1× 3090	2× 3090	4× 3090
Decode, 4K prompt	124 → 125 (+1.1 %)	207 → 210 (+1.9 %)	250 → 251 (+0.1 %)
Decode, 131K prompt	95 → 95 (+0.4 %)	173 → 182 (+5.1 %)	235 → 243 (+3.1 %)
Decode, 262K prompt	73 → 71 (−2.1 %)	142 → 143 (+0.7 %)	215 → 238 (+11.0 %)
Decode, 262K, thinking off	49 → 50 (+2.4 %)	111 → 109 (−2.1 %)	158 → 149 (−5.7 %)
Cold prefill, 10,157 tokens	1,021 → 1,200 (+17.5 %)	1,823 → 2,301 (+26.2 %)	3,009 → 3,866 (+28.5 %)
Cold prefill, 101,845 tokens	708 → 871 (+23.0 %)	1,297 → 1,672 (+28.9 %)	2,295 → 2,995 (+30.5 %)
Cold prefill, 261,888 tokens	456 → 579 (+27.0 %)	858 → 1,115 (+29.9 %)	1,596 → 2,101 (+31.6 %)
8 agents, thinking off	n/a	517 → 615 (+18.9 %)	752 → 854 (+13.6 %)

Host: 4× RTX 3090 (24 GB, sm_86) at 220 W, PCIe Gen4 with P2P, no NVLink, driver 610.43.02; the 3.0.1 runs also with the optional Zagitta's gpu-unlock host upgrade.
Engine: Merlin 3.0.1 (the engine wheel is unchanged from 3.0.0), served by merlin serve with the shipped defaults, one fresh boot per layout. One card: TP1; two cards: TP2 on GPUs 0 and 2; four: TP4. Runs 2026-11-01 to 2026-11-02, on release candidates just before the final commit, not on the final commit itself.
Layout: MTP K=4, FP8 KV, RecoverSSM on every layout, host-resident token embeddings, thinking effort medium (the server default).
Decode: T=0, a needle prompt at depth, every send a cold prefill except sends 2 and 3 at 262K, which reuse the prefix before the needle; one discarded warm-up, then 3 sends per cell, median. tok/s = tokens after the first streamed event / time from first to last event.
Streams are client-timed. Full tables, thinking-off rows and ITL: the bench card.

Quality

This checkpoint, measured on the Merlin engine:

Table with columns: Check, Result, Setting
Check	Result	Setting
GSM8K-200	195 / 194 / 197 of 200 on 1 / 2 / 4 cards	5-shot, thinking off
GSM8K-200, thinking on	198 / 200 on four cards	5-shot, effort medium, none cut at the 8,192-token cap
Needles at 131K	5 / 5 on each layout	single-occurrence needles, an image in the prompt, thinking off, greedy, cold cache
Needles at 262K	12 / 12 on each layout	as above
Needle at exactly 261,888 tokens	3 / 3 on each layout	thinking off
Image, 4096 × 2048 (8,388,608 px)	1 / 1 · 2 / 2 at once · 1 / 1 with four in one prompt, on 2 and 4 cards	thinking off; names both colours of a two-colour image (with four, of the last one)

The base model, as reported by Qwen for the BF16 weights:

Table with columns: Area, Benchmark, Qwen3.8-27B (BF16)
Area	Benchmark	Qwen3.8-27B (BF16)
Coding	Terminal Bench 2.1 (Terminus)	73.0
Coding	SWE-bench Pro	61.7
Coding	LiveCodeBench v6	90.3
Agents	CoWorkBench	70.7
Reasoning and instructions	GPQA Diamond	89.2
Reasoning and instructions	HLE	30.8
Reasoning and instructions	IFBench	79.5
Vision	OSWorld-Verified	84.3
Vision	AndroidWorld	81.9
Vision	RealWorldQA	85.9

A selection from the upstream card, which has the full tables and comparisons with other models. These were not re-run on this 4-bit build.

What it fits on

Table with columns: Measured on, Context per request, KV pool, Held at once, Images
Measured on	Context per request	KV pool	Held at once	Images
1× RTX 3090, 24 GB (🟢 Tested)	262,144 tokens	271,506 tokens	1 × 262K or 2 × 131K	1 per prompt, 4 MP
2× RTX 3090, 24 GB (🟢 Tested)	262,144 tokens	826,689 tokens	3 × 262K or 6 × 131K	8 MP
4× RTX 3090, 24 GB (🟢 Tested)	262,144 tokens	2,012,105 tokens	7 × 262K or 14 × 131K	8 MP

Other GPUs. The engine also ships profiles for these; the Status column says how far each has been run:

Table with columns: GPUs, Arch, Profile, Context fit, Status
GPUs	Arch	Profile	Context fit	Status
2 / 4× RTX 3090 without P2P	Ampere, sm_86	`27b-tp2-3090-nop2p`, `27b-tp4-3090-nop2p`	~3 / ~7 × 262K (the 3090 pins)	⚫ Untested
1 / 2 / 4× RTX 3090 Ti, RTX A5000 (24 GB)	Ampere, sm_86	`27b-tp1-3090`, `-tp2-3090`, `-tp4-3090`	~1 / ~3 / ~7 × 262K (as the 3090)	⚫ Untested
1 / 2 / 4× RTX 4090 (24 GB)	Ada, sm_89	`27b-tp1-sm89-24g`, `-tp2-`, `-tp4-`	~1 / ~3 / 7 × 262K (four cards: 7 × 262K held on a rented host with Qwen3.8-27B; one card: the full run ran out of memory at chunk 1024; at the shipped chunk 512 only the 131K and 262K needles have run)	🟡 Testing on 2×, 4×; ⚫ otherwise
1 / 2 / 4× 32 / 48 GB Ada: RTX 4080 Super (modified), RTX 4090 48 GB (modified), RTX 6000 Ada, L40S	Ada, sm_89	`27b-tpN-sm89-32g`, `-sm89-48g`	~1 / ~3 / ~7 × 262K (the 24 GB pins)	⚫ Untested
1 / 2 / 4× 48 GB: RTX A6000, A40	Ampere, sm_86	`27b-tpN-sm86-48g`	~3 / ~8 / ~18 × 262K	⚫ Untested
1 / 2 / 4× A30 (24 GB)	Ampere, sm_80	`27b-tpN-sm80-24g`	~1 / ~3 / ~7 × 262K	⚫ Untested
1 / 2 / 4× CMP 170HX 32 GB	Ampere, sm_80	`27b-tpN-sm80-32g`	~1 / ~4 / ~11 × 262K	⚫ Untested
1 / 2 / 4× A100 40 GB	Ampere, sm_80	`27b-tpN-sm80-40g`	~2 / ~6 / ~14 × 262K	⚫ Untested
1 / 2 / 4× CMP 170HX 64 GB	Ampere, sm_80	`27b-tpN-sm80-64g`	~5 / ~11 / ~25 × 262K (needles exact on a rented host)	🟡 Testing
1 / 2 / 4× A100 80 GB	Ampere, sm_80	`27b-tpN-sm80-80g`	~7 / ~15 / ~32 × 262K	⚫ Untested

Table with columns: Need, Details
Need	Details
Host RAM	About 2.4 GiB pinned for the host-resident token embeddings (1.18 GiB per card on two cards, 0.59 on four), on top of normal use; one card also keeps the vision-tower weights in host RAM (about 1 GiB)
KV RAM cache (optional)	Off by default. `--kv-ram-cache SIZE` pins SIZE of host RAM on top of the RAM above, at least 9 GiB on one card, 10 on two and 12 on four; see Use it
Disk	About 16 GiB for the checkpoint, plus the engine environment and caches
Engine host	Linux x86-64 (glibc 2.38 or newer), NVIDIA driver R580 or later (CUDA 13), Python 3.13, GCC/G++ 13 or newer, `/dev/shm` of at least 1 GB; GPUs headless with no other CUDA process. `merlin doctor` checks the host.

Use it

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine./setup.sh --model 27b

git clone https://github.com/halt95/merlin-engine.git && cd merlin-engine./setup.sh --model 27b

[!IMPORTANT] This checkpoint needs the Merlin engine; stock vLLM was not tested with it.

bash

git clone https://github.com/halt95/merlin-engine.git && cd merlin-enginecurl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whlMERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" \  bash release/install-env.sh ./venv venv/bin/hf download halt95/Qwen3.8-27B-W4A16-Merlin --revision 7a0af39f63dd4fb0733c23d3ec126762074a503f \  --local-dir ~/models/27b(cd ~/models/27b && sha256sum -c "$OLDPWD/models/qwen38-27b/checkpoint.sha256")venv/bin/python merlin_cli.py serve ~/models/27b

git clone https://github.com/halt95/merlin-engine.git && cd merlin-enginecurl -LO https://github.com/halt95/merlin-engine/releases/download/merlin-v3.0.1/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whlMERLIN_WHEEL="$PWD/vllm-0.32.0+merlin.7002f56537-cp313-cp313-linux_x86_64.whl" \  bash release/install-env.sh ./venv venv/bin/hf download halt95/Qwen3.8-27B-W4A16-Merlin --revision 7a0af39f63dd4fb0733c23d3ec126762074a503f \  --local-dir ~/models/27b(cd ~/models/27b && sha256sum -c "$OLDPWD/models/qwen38-27b/checkpoint.sha256")venv/bin/python merlin_cli.py serve ~/models/27b

Recommended settings (Qwen's, for Qwen3.8-27B):

Table with columns: Mode, temperature, top_p, top_k, presence_penalty
Mode	temperature	top_p	top_k	presence_penalty
Thinking (default)	1.0	0.95	20	0.0
Non-thinking	0.7	0.80	20	1.5

Qwen's recommendation for this model, with min_p=0.0 and repetition_penalty=1.0 in both modes. A presence_penalty between 0 and 2 reduces endless repetition; higher values can mix languages.

Turn thinking off for a request with "chat_template_kwargs": {"enable_thinking": false}. With thinking on, a short max_tokens can end inside the reasoning with empty content.
Reasoning effort: Merlin serves medium by default; pick another per request with "chat_template_kwargs": {"reasoning_effort": "low"} (low / medium / xhigh).
KV RAM cache (experimental, off by default): --kv-ram-cache SIZE on merlin serve or ./setup.sh keeps the prefix KV that the GPU evicts in pinned host RAM and restores it, with the same output, instead of recomputing it. It adds no context or sessions; SIZE comes on top of the host RAM above (at least 9g on one card, 10g on two, 12g on four). On four cards --swarm raises the session cap from 24 to 32 and sets a 48g cache. Under heavy concurrent prefill a session may recompute instead of restoring (output identical).
--spec dflash2 is the Flash-Next family's drafter option; it is refused on this checkpoint, which runs its own MTP head.
Chat template: Merlin serves Froggeric's fixed Qwen3.8 template (v22.5) by default, not the file in this repository. It accepts system messages after the first turn, so coding agents' per-turn notes stay in place and the prefix cache keeps hitting. CHAT_TEMPLATE=<path> serves another template; CHAT_TEMPLATE= (empty) serves this repository's own.
With Qwen's own template (CHAT_TEMPLATE=), earlier turns' reasoning is kept by default and "chat_template_kwargs": {"preserve_thinking": false} keeps only the latest.
Images: up to 64 per prompt at 8 MP each on two or four cards; one card takes one image per prompt at 4 MP. The cap is a served default: a client that raises it per request goes past the measured headroom.
Tool calls use the standard tools field, parsed with the Qwen3 coder tool parser.
To reach it from other machines, answer network at setup's Access question (it generates an API key), or start merlin serve with HOST=0.0.0.0 and VLLM_API_KEY set; put a TLS reverse proxy in front before opening it beyond your own network.

The base model

Qwen3.8-27B is Qwen's dense model on the Qwen3.5 architecture, with vision. The architecture, in short:

Hybrid attention. Three Gated DeltaNet (linear-attention) layers for every gated-attention layer: 48 and 16 of the 64 layers.
Dense. All 27B parameters work on every token; no experts.
Multi-token prediction. One MTP layer, trained over several steps; Merlin uses it for speculative decoding (4 draft tokens).
Thinking by default, with a non-thinking mode per request and reasoning_effort levels; Qwen's template keeps earlier turns' reasoning by default (preserved thinking).

Table with columns: Property, Qwen3.8-27B
Property	Qwen3.8-27B
Type	Causal language model with a vision encoder (images and video)
Parameters	27B
Layers	64: 16 × (3 × Gated DeltaNet → FFN, then 1 × gated attention → FFN)
Attention	24 query / 4 KV heads, head dim 256 · Gated DeltaNet: 48 V / 16 QK heads, head dim 128
Feed-forward	Width 17,408
Context	262,144 tokens native (Qwen: extensible to 1M; this build serves 262,144)

Architecture as reported by Qwen on the upstream card.

Other builds

Table with columns: Checkpoint, Base, GPUs, Decode at 262K
Checkpoint	Base	GPUs	Decode at 262K
Qwen3.8-27B (this card)	Qwen3.8-27B, dense	1×, 2× or 4× 3090	71 / 143 / 238 tok/s
Flash-Next	Qwen3.8-Flash-Next, 125B MoE	4× 3090	197 tok/s
Swift1.5	Flash-Next fine-tune	4× 3090	186 tok/s
Swift1.5-Huihui	Swift1.5, abliterated	4× 3090	196 tok/s
ThinkingCap	Qwen3.8-27B fine-tune	2× or 4× 3090	160 / 238 tok/s
Nemotron 3.5 Super VL	Nemotron 3.5 Super VL, 120B-A12B MoE	4× 3090	93 tok/s
Ornith 1.5	Qwen3.5-122B-A10B fine-tune (ornith-ai), 122B MoE	4× 3090	105 tok/s

Good to know

Every figure we measured for this checkpoint comes from 4× RTX 3090 at 220 W, PCIe Gen4 with P2P, used as one, two or four cards, except the rental rows under Other GPUs; the 3.0.1 figures were also measured with the optional GPU-unlock host upgrade, the 3.0.0 ones were not; the Other builds table names each build's card count.
The same engine profiles serve ThinkingCap, BottleCap AI's fine-tune of this model, which reasons in fewer tokens (under a different licence).
Coming from the qwen38-27b-3090s engine (v1.x)? It is replaced by the Merlin engine; this checkpoint's files are unchanged.

Changes

Table with columns: Date, Change to this repository
Date	Change to this repository
2026-11-02	Card updated for Merlin 3.0.1 (speed figures, install tag, KV RAM cache note). The weights, config and tokenizer are unchanged
2026-10-12	`tokenizer.json` replaced with the upstream Qwen3.8-27B file: combining-mark scripts (Hindi, Thai, Arabic and others) now tokenize as upstream with transformers 5.18.0 or newer (which Merlin pins; 5.17.0 and earlier ignore this file); English, code and CJK are unchanged
2026-09-27	Weights published; unchanged since

Other card-only edits are not listed. Engine releases and their speed changes: the engine's CHANGELOG.