halt95

Qwen3.8-27B-W4A16-Merlin

README

License: apache-2.0

Serving

Use the serving repository, version v1.0.0: halt95/qwen38-27b-3090s. It installs stock vLLM 0.30.0 from PyPI plus five Python patches (hash-locked, nothing compiled at install) and serves this checkpoint at 262K context on dual RTX 3090s (TP=2) or quad (TP=4), with MTP K=3 and FP8 KV. Validated on sm_86 (Ampere) only.

bash

git clone --branch v1.0.0 https://github.com/halt95/qwen38-27b-3090s && cd qwen38-27b-3090srelease/install-env.sh .venvHF_HUB_DISABLE_TELEMETRY=1 .venv/bin/hf download halt95/Qwen3.8-27B-W4A16-Merlin --local-dir ./ckpt(cd ckpt && sha256sum -c ../release/checkpoint.sha256)MODEL=$PWD/ckpt serve/serve.sh          # serve/serve-tp4.sh for quad card

It is working when the log reads GPU KV cache size: 794,351 tokens (1,949,844 on quad card; with SPEC_K=1 the same pins give 810,191 and 1,987,859; the pool is pinned in bytes, so any other number means a different setup) and Application startup complete, and a one-token generation on 127.0.0.1:8100 (model qwen38-27b) answers. Needs Linux x86_64, Python 3.13, a driver for CUDA 13.0, nvcc for FlashInfer's JIT, and about 2.4 GiB of extra system RAM: the serving stack's own host-resident embedding patch keeps the token-embedding table in pinned host RAM (1.184 GiB per GPU on dual card, 0.592 on quad card) and gives that VRAM to the KV cache. Full requirements, troubleshooting and every knob are in the repository's README. Both serve scripts expect working GPU peer-to-peer (P2P).

Measured (v1.0.0, reference host)

Table

KV pool	794,351 tokens on dual card (3 × 262K sessions at once); 1,949,844 on quad card (7 × 262K)
single-stream decode step (median ITL), dual card, 4K / 32K / 131K / 250K	19.8 / 21.5 / 26.8 / 33.5 ms, 3.4–3.5 tokens per step
same, quad card	15.3 / 16.2 / 18.7 / 21.8 ms
single-stream decode, 4K / 32K / 131K / 250K	171.4 / 159.3 / 132.3 / 108.3 tok/s dual card; 231.2 / 210.8 / 190.8 / 165.5 quad card
decode throughput, dual card, 1 / 2 / 4 / 8 / 32 streams (≈54-token prompts, 1,024 out)	119.6 / 227.1 / 393.2 / 520.6 / 771.7 tok/s with K=3 (default); 106.6 / 208.4 / 390.1 / 644.3 / 946.4 with `SPEC_K=1` (on two card pairs that differ by about 5 %)
decode throughput, quad card, 1 / 8 / 32 streams	148.9 / 701.9 / 1,120.9 tok/s with K=3; 150.4 / 849.6 / 948.1 with

Qwen3.8-27B v1.0.0 over context depth, single stream: cold prefill 1,844 to 900 tok/s on dual card and 2,976 to 1,608 on quad card; decode 171 to 108 tok/s and 231 to 165 tok/s; decode step 19.8 to 33.5 ms and 15.3 to 21.8 ms, 4K to 250K

Quality. Paired GSM8K-1319, same checkpoint, v1.0.0 against an earlier three-patch build of the serving stack (thinking off, 512-token cap): 94.69 % vs 94.47 %, Δ +0.23 pp, 95 % CI [−0.96, +1.42] pp; the non-inferiority margin, −2 pp, was fixed before the run: pass. Repetition loops: 0 genuine collapses in 108 long generations on v1.0.0 and 0 in 504 on earlier builds (including one with a BF16 head).

The bench cards are in the repository under benchmarks/.

How it was made

The lm_head and the MTP drafter head were quantized with GPTQ using input Hessians captured from the served model; the drafter head was calibrated on drafter inputs. Codes are packed with compressed-tensors pack_to_int32 in the signed range.
FP8 E4M3 KV scales were calibrated per full-attention layer from observed absolute maxima, with a 1.10 margin, on the philbert checkpoint before the head and drafter were quantised; that traffic included the first 50 GSM8K test questions (activation maxima only).
Calibration data: public benchmark prompts (ARC, MMLU-Pro, GPQA, GSM8K train split, HumanEval), template-generated synthetic tasks and a small hand-written prompt set, together with the model's own responses to them. No calibration data is distributed.
The tools are in the serving repository under quant/. The AWQ recipe for the base weights is philbert's; see philbert440/Qwen3.8-27B-W4A16-AWQ.

Known behaviours and limits

All measurements are from one reference host (RTX 3090s, PCIe Gen4 x16, P2P-enabled driver, 220 W cap).

sm_86 only. Other GPUs, non-P2P hosts and narrower PCIe links are unmeasured.
Quad card: v1.0.0 measurements (performance at the earlier 2,010,034-token pin; the shipped pin re-verified), GSM8K-1319 paired against dual card 94.47 % vs 94.69 % (margin −2 pp: pass); its soaks ran on earlier builds.
With vLLM 0.30 + MTP, prefix-cache reuse covers prompt prefixes, not a previous turn's generated tokens (a recompute cost, not a correctness issue).
A request reusing prefix-cache blocks can differ from a cold one at near-tied tokens; greedy output repeats for identical requests.
The base model can loop occasionally; none was seen in the v1.0.0 loop check.
Soak-tested on earlier builds: 8 h on dual card (3,808 requests) and 2 h on quad card (999 requests) on the three-patch build, 0 errors; the pre-release five-patch build 65 min / 35 min, 0 errors. No full-length soak on v1.0.0 itself.

Licence and attribution

Apache-2.0, like the base model and the AWQ checkpoint. Credit to Qwen (Alibaba Cloud) for Qwen3.8-27B and to philbert for the AWQ checkpoint philbert440/Qwen3.8-27B-W4A16-AWQ. The serving stack is halt95/qwen38-27b-3090s (Apache-2.0); its NOTICE credits vLLM, vLLM PR #47979, FlashInfer, compressed-tensors and froggeric's chat template; this repository's own NOTICE covers the checkpoint. Sibling release for the MoE model: halt95/qwen38-flash-next-3090s.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

halt95

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality