Capicua25x

Qwen3.8-27B-MXFP4-Quark-RDNA4

README

License: apache-2.0

Quick start (2× R9700, TP2, full native 262k window, MTP-3)

bash
docker run --rm --name vllm-qwen --network=host \
  --device=/dev/kfd --device=/dev/dri/renderD128 --device=/dev/dri/renderD129 \
  --group-add=video --group-add=render --ipc=host \
  -e NCCL_PROTO=Simple \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint /usr/local/bin/vllm capicua25x/vllm-rocm-rdna4:0.26.1-rdna4-rc10 \
  serve Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4 --served-model-name qwen --port 8011 --trust-remote-code \
  --tensor-parallel-size 2 --gpu-memory-utilization 0.95 --max-model-len 262144 \
  --attention-backend TRITON_ATTN --enable-prefix-caching \
  --max-num-seqs 32 --max-num-batched-tokens 8000 --max-cudagraph-capture-size 128 \
  --enable-auto-tool-choice --tool-call-parser qwen3_xml --reasoning-parser qwen3 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"attention_backend":"TRITON_ATTN"}'

We A/B this configuration (C · MXFP4 @ bf16 KV) in production against B · FP8 @ fp8 KV (Qwen/Qwen3.8-27B-FP8 +

markdown

--kv-cache-dtype fp8 --mamba-ssm-cache-dtype bfloat16

). Both pass the same gates; so far Qwen behaves correctly on both.

Thinking mode — read this before benchmarking

Qwen3.8's reasoning ("thinking") is controlled by a chat-template kwarg, not by the OpenAI reasoning_effort parameter. vLLM silently ignores reasoning_effort — requests that pass it run in no-think mode with no error, which measurably degrades agentic/multi-step performance (we measured τ²-Bench telecom dropping from ~0.90 to ~0.63 on both this model and the FP8 arm before catching it). To enable thinking, send:

json
{"chat_template_kwargs": {"enable_thinking": true}}

at the top level of the request body (or inside extra_body when using an OpenAI client / LiteLLM). With --reasoning-parser qwen3 (as in the quick start), the think block is stripped into reasoning_content and the visible content stays clean — tool calls are unaffected. A cheap preflight to verify thinking is active: send a short question with max_tokens: 60; a thinking serve spends most of the budget invisibly (tiny visible content vs completion_tokens), a no-think serve returns ~60 tokens of visible prose. All accuracy numbers above were measured with thinking ON except the "nothink" rows.

Measured performance (per-user / aggregate tok/s)

Paired back-to-back on the same box (2× R9700, TP2, thinking ON, 256-token completions). C = this model @ bf16 KV · B = FP8 checkpoint @ fp8 KV.

Note (2026-08-24): the two C-vs-B tables below are pre-v4, replay-era measurements — valid as comparatives (both arms ran the identical replay-prone workload back-to-back), but the absolute essay tok/s values are inflated by the fixed-prompt/temp-0 replay confound (shape-dependent on re-measurement: ~10-20% on most greedy-raw cells, up to ~2x on the worst (a GB10-pair short cell), while one sampled-path 6k cell even measured slightly higher under v4 (different sampling provenance + build drift)) described in the measurement update above. Honest v4 absolutes follow in the third table.

Short prompts (~30 tokens — interactive chat) — pre-v4, replay-era: comparative only (see note)

Table with columns: users, C (this model), B
users	C (this model)	B
1	62 / 62	66 / 66
4	53 / 208	60 / 232
8	42 / 335	54 / 421
16	32 / 505	45 / 687
32	24 / 713	33 / 1,014
64	17 / 714

B column: in-tree tuned R9700 GEMM configs; C column: measured on rc10 (both 2026-08-20).

6,000-token prompts (RAG / long-system-prompt workloads) — pre-v4, replay-era: comparative only (see note)

Table with columns: users, C (this model), B
users	C (this model)	B
1	54 / 54	61 / 61
4	43 / 163	43 / 170
8	30 / 231	31 / 241
16	20 / 312	20 / 311
32	12 / 368	11 / 356

At long context the two are a statistical tie; on short prompts B leads from 8 users up. KV pool: C ≈ 415k tokens · B ≈ 539k (2.06× the window). NCCL_PROTO=Simple matters on this PCIe pair (RCCL's LL protocol is 2.8× slower for the ~640 KB decode all-reduces).

Honest v4 absolutes (measured 2026-08-24, idle-verified) — FP8 arm B, rc10, MTP-3 (k=3)

Bench v4: rotating distinct topics + per-invocation nonce, temp 0.7 / top_p 0.95 (closer to production sampling than the old greedy fixed prompt — nothing is ever regenerated). Per-user / aggregate tok/s; acceptance is accepted-per-draft out of k=3, scraped from /metrics.

Table with columns: workload, c1, c4, c16, c32, c64, acceptance (of 3)
workload	c1	c4	c16	c32	c64	acceptance (of 3)
short essay	59.1 / 59	49.5 / 183	37.1 / 541	27.0 / 789	20.1 / 825	1.52 @ c1
6k-prefix essay	55.4 / 55	44.1 / 168	21.8 / 327	12.8 / 390	8.4 / 397

6k aggregate saturates at ~390–400 from c=32 (c48: 10.0 / 373). Comfort ceiling at ≥20 tok/s per user: ~16 in-flight at 6k, ~64 on short prompts. Novel prose lands ~2.4–2.8 tokens/step regardless of draft length. The MXFP4 arm (C, this model) has not been re-run under v4 yet — its pre-v4 essay absolutes above carry the same class of inflation; discount them accordingly and use the C-vs-B tables for the relative shape only.

Accuracy (AA class-A, paired items, seed 1234, on-spec sampling)

ref = the same checkpoint served in bf16 by a cloud provider. Same judge for all judged rows. Cells show the most recent run at the stated n on the shipping config; ±2 items is the noise band.

Table with columns: benchmark (n), ref (bf16), C (this model), B
benchmark (n)	ref (bf16)	C (this model)	B
GSM8K think, flex·strict (50)	0.96·0.82	0.98·0.96	0.98·0.86
GSM8K nothink (50)	0.98·0.98	0.98·0.98	0.98·0.98
IFEval inst·prompt (80)	0.97·0.95	0.95·0.91	0.98·0.98
AA-LCR judged (100)	0.78	0.78	0.77 · 0.81 (s1234·s99)

Reproducing the quantisation

Data-free, CPU-only, file-to-file — ~3 minutes; no calibration set. (The KV-cache scalars are the one calibrated artefact and came from a separate capture pass on the served model.)

python
from quark.torch.export.api import direct_quantize_checkpoint

EXCLUDE = [
    "lm_head", "*embed_tokens*",
    "*.self_attn.q_proj", "*.self_attn.k_proj", "*.self_attn.v_proj", "*.self_attn.o_proj",
    "*.self_attn.q_norm", "*.self_attn.k_norm", "*norm*",
    "*.linear_attn.conv1d", "*.linear_attn.norm",
    "*.mlp.gate", "*.mlp.shared_expert_gate",
    "mtp*", "*visual*", "*vision*",
]

Full recipe, engine patches and methodology: RDNA4-PORT.md · throughput numbers are reproducible with Capicua25x/modelbench (bench v4; the pre-v4 replay-era essay absolutes are not reproducible by design — see the 2026-08-24 measurement update).

Licence and attribution

Apache-2.0, following the base model. Quantised and served by Capicua25x; base model by the Qwen team; quantisation toolkit by AMD (Quark).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Capicua25x

Model Tree

Base

Qwen/Qwen3.8-27B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality