Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB API & Inference Endpoint

What is in this repository

Which one to use

All Qwopus3.8-27B-Flash-V2 variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Table with columns: Variant, Weights, Context on one RTX 3090, KLD vs BF16, Agent tasks, Decode, 1 request, Pick it for
Variant	Weights	Context on one RTX 3090	KLD vs BF16	Agent tasks	Decode, 1 request	Pick it for
GPTQ INT4 (this repo)	14.21 GiB	222,000 (text) · 176,128 with vision	0.0357	29/30	114.3 tok/s	the longest context on stock vLLM (+ a 4-line patch): 222,000 text-only, or 176,128 with vision
GPTQ INT8 g64	15.75 GiB	176,000 (text) · 131,072 with vision	0.0145	25/30	97.3 tok/s	a more faithful GPTQ (KLD 0.0145) on stock vLLM (+ the same patch), with less context
EXL3 3.85	13.62 GiB	215,000 with vision	0.0060	27/30	86.9 tok/s	the closest to BF16 (KLD 0.0060) with vision at 215,000; needs exllamav3 + our plugin
EXL3 3.70	13.19 GiB	226,000 with vision	0.0073	25/30	83.9 tok/s	more context (226,000) and 4 sessions with vision, a little less faithful than 3.85

Variants / revisions

Table with columns: Revision, MTP head, Size, Notes
Revision	MTP head	Size	Notes
`main`	BF16 (0.79 GiB)	14.6 GiB	reference; use without speculation for the full 262k context
`mtp-int8`	INT8 g128 (0.40 GiB)	14.21 GiB	recommended with MTP: same acceptance as BF16, +≈0.4 GiB for KV cache

Files and reproducibility

scripts/quantize.py: exact quantization script (llm-compressor); recipe.yaml: the recipe as serialized by llm-compressor.
scripts/mtp_quant.py: script that builds the mtp-int8 revision from main.
config.json.orig-llmcompressor: config as emitted by llm-compressor, before the two vLLM fixes described above.
chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
patches/: vLLM patch and its Apache-2.0 license.
SHA256SUMS: checksums of the model files.

Results

Family comparison (Qwopus3.8-27B-Flash-V2, same conditions)

Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (GPTQ: + the embedding patch / EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).

Table with columns: Variant, Weights, KLD mean / median (internal · public), Perplexity (BF16 1.6706 · pub. 1.4288), Needle (8 facts) near max. context, Tools: real opencode request, Agent tasks (10 × 3), Reasoning xhigh: correct · tokens, Vision
Variant	Weights	KLD mean / median (internal · public)	Perplexity (BF16 1.6706 · pub. 1.4288)	Needle (8 facts) near max. context	Tools: real opencode request	Agent tasks (10 × 3)	Reasoning `xhigh`: correct · tokens	Vision
GPTQ W4A16 + INT4 embeddings (`mtp-int8`, this repo)	14.21 GiB	0.0357 / 0.0129 · pub. 0.0251 / 0.0024	1.7436 · pub. 1.4707	8/8 at 204,551	55/60	29/30	47/48 · 18,090	15/15
GPTQ W4A16 g64 + INT8 embeddings (`main`)	15.75 GiB	0.0145 / 0.0047 · pub. 0.0115 / 0.0011	1.7029 · pub. 1.4507	8/8 at 162,208	57/60	25/30	48/48 · 17,965	15/15
EXL3 3.85 bpw (`main`)	13.62 GiB	0.0060 / 0.0018 · pub. 0.0048 / 0.0004	1.6815 · pub. 1.4379	8/8 at 197,991	54/60	27/30	48/48 · 23,460	15/15
EXL3 3.70 bpw (`3.70bpw`)	13.19 GiB	0.0073 / 0.0022 · pub. 0.0060 / 0.0005	1.6856 · pub. 1.4409	8/8 at 208,158	57/60	25/30	48/48 · 26,996	15/15

Table with columns: Variant, Context on one RTX 3090, Sessions, Prefill near max. context, Decode, 1 / 2 / 8 requests (tok/s), MTP tokens per step, VRAM really free at peak
Variant	Context on one RTX 3090	Sessions	Prefill near max. context	Decode, 1 / 2 / 8 requests (tok/s)	MTP tokens per step	VRAM really free at peak
GPTQ INT4	222,000 (text) · 176,128 with vision	2	603 tok/s	114.3 / 192.2 / 189.5	3.20	284 MiB · vision config: 930 MiB
GPTQ INT8 g64	176,000 (text) · 131,072 with vision	2	644 tok/s	97.3 / 173.0 / 175.1	3.21	196 MiB · vision config: 840 MiB
EXL3 3.85	215,000 with vision	2	554 tok/s	86.9 / 140.5 / 145.3	3.40	320 MiB
EXL3 3.70	226,000 with vision	4	543 tok/s	83.9 / 145.7 / 258.8	3.21	148 MiB

How to read it:

KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign.
Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test.
Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run.
VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB.
Older sections below keep their own date and setup; use this table to compare variants.

How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):

Table with columns: Variant, KV pool (tokens), Max sessions, 32k each, 64k each, 128k each, Full context each
Variant	KV pool (tokens)	Max sessions	32k each	64k each	128k each	Full context each
GPTQ INT4	223,441	2	2	2	1	1 × 222,000
GPTQ INT4 (vision config)	178,923	2	2	2	1	1 × 176,128
GPTQ INT8 g64	178,816	2	2	2	1	1 × 176,000
GPTQ INT8 g64 (vision config)	132,423	2	2	2	1	1 × 131,072
EXL3 3.85	216,433	2	2	2	1	1 × 215,000
EXL3 3.70	231,757	4	4	3	1	1 × 226,000

Few users or many users. The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.

Table with columns: Variant, Configuration, Decode by simultaneous requests (aggregate tok/s), N sessions at once, needle 8/8, Tools (5 × 3), VRAM really free
Variant	Configuration	Decode by simultaneous requests (aggregate tok/s)	N sessions at once, needle 8/8	Tools (5 × 3)	VRAM really free
GPTQ INT4	Quick start (2 sessions, MTP 3)	1: 114 · 2: 192 · 8: 190	—	55/60	284 MiB
GPTQ INT4	many users (8 sessions of up to 222,000 tokens, MTP 3)	1: 113 · 2: 190 · 4: 340 · 8: 481	8/8 of 22k	15/15	174 MiB
GPTQ INT8 g64	Quick start (2 sessions, MTP 3)	1: 97 · 2: 173 · 8: 175	—	57/60	196 MiB
GPTQ INT8 g64	many users (8 sessions of up to 165,000 tokens, MTP 3)	1: 96 · 2: 166 · 4: 316 · 8: 460	8/8 of 22k	14/15	386 MiB
EXL3 3.85	Quick start (2 sessions, MTP 3)	1: 87 · 2: 140 · 8: 145	—	54/60	320 MiB
EXL3 3.85	many users (8 sessions of up to 215,000 tokens, MTP 3)	1: 88 · 2: 142 · 4: 257 · 8: 314	8/8 of 22k	14/15	138 MiB
EXL3 3.70	Quick start (4 sessions, MTP 3)	1: 84 · 2: 146 · 8: 259	—	57/60	148 MiB
EXL3 3.70	many users (8 sessions of up to 220,000 tokens, MTP 3)	1: 78 · 2: 143 · 4: 254 · 8: 293	8/8 of 22k	15/15	358 MiB

Benchmarks and validated configurations (single RTX 3090 24 GB)

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

Setup: vLLM 0.29.0 + patch, CUDA 13.0 wheels, driver 595 (CUDA 13.2), fp8 (e4m3) KV cache without calibrated scales, FlashInfer, --language-model-only, 370 W power limit (350 W for the 2026-10-02 runs). Decode is measured with 800 forced output tokens (ignore_eos).

Long context: 8 distinct facts spread from 3 % to 97 % depth, all 8 must be returned (greedy)

Table with columns: Configuration, Prompt tokens, Recovered, Time, Prefill
Configuration	Prompt tokens	Recovered	Time	Prefill
No MTP, 262k	57,874	8/8	59.8 s	968 tok/s
No MTP, 262k	246,276	8/8	435.2 s	566 tok/s
MTP 1 token (BF16 head), 208k	199,086	8/8	326.5 s	610 tok/s
MTP 3 tokens (INT8 head), 200k, prefix caching	189,270	8/8	306.4 s	618 tok/s

Decode speed and MTP (Qwopus MTP head)

Table with columns: Configuration, Context, 1 request, t=0 / t=0.6, 4 concurrent, aggregate, Accepted tokens per step, Per-position acceptance
Configuration	Context	1 request, t=0 / t=0.6	4 concurrent, aggregate	Accepted tokens per step	Per-position acceptance
No speculation	262k	49.9 / 50.5 tok/s	177 tok/s	—	—
MTP, 1 token	208k	76.5 / 72.6 tok/s	287 tok/s	1.85	85 %
MTP, 2 tokens	208k	88.3 / 84.8 tok/s	304 tok/s	2.43	82 / 62 %
MTP, 3 tokens (BF16 head)	200k	95.5 / 89.9 tok/s	313 tok/s	≈2.98	83 / 64 / 50 %
MTP, 3 tokens (INT8 head)	200k	95.5 / 95.8 tok/s	168 tok/s (2 seqs)	≈2.91	83 / 63 / 46 %
MTP, 3 tokens (INT4 head, not published)	200k	94.8 / 92.3 tok/s	174 tok/s (2 seqs)	2.87	83 / 61 / 43 %

Speculative decoding with MTP does not change the output: every drafted token is verified by the main model. Prefill cost of MTP: −1 to −3 %.

Prefix caching in an agentic session (≈90k-token history, each turn appends ≈2k tokens of tool output; time to first token)

Table with columns: Turn, 0, 1, 2, 3, 4, 5, 6
Turn	0	1	2	3	4	5	6
Without prefix caching	98.5 s	103.5 s	108.0 s	112.8 s	117.4 s	122.1 s	126.9 s
With `--enable-prefix-caching`	100.5 s	5.7 s	5.7 s	5.6 s	5.4 s	5.2 s	7.5 s

KV capacity cost of prefix caching: −2 % (270,663 → 265,356 tokens). With MTP (INT8) enabled as well, later turns take ≈6.8 s.

Concurrency (4 × 58k-token prompts at once, no MTP): 8/8 retrieval in all four, 988 tok/s aggregate prefill (same efficiency as one request).

VRAM (peak, 24,576 MiB total): 23,816 MiB without MTP (262k, 4 seqs); 23,592 MiB with MTP-INT8 (200k, 2 seqs, prefix caching).

Agent turns and 8 sessions (2026-10-02, 350 W, text, MTP 3; agent turn = the real opencode request, ≈13.5k → 14.5k tokens over 10 turns, with prefix caching; 8 sessions = --max-num-seqs 8, the KV pool shared by all):

Table with columns: Configuration, Turn: time to first token / total (median), First turn (no cache), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0 / 0.6), Really free (min)
Configuration	Turn: time to first token / total (median)	First turn (no cache)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0 / 0.6)	Really free (min)
150,000, KV 6.25e9	2.6 / 3.3 s	11.6 s	8 × 16.5k, 8/8 in all, 1,129 tok/s	462.8 / 447.3 tok/s	2,571 MiB
200,000, KV 8.15e9	2.6 / 3.3 s	11.6 s	8 × 22k, 8/8 in all, 1,105 tok/s	459.5 / 432.7 tok/s	751 MiB

Tool calling with a real coding-agent request (the actual request of the opencode agent: long system prompt, 11 tools; three cases — a fresh "list the files" request, the same after a long story, and a full multi-turn session — 5 tries each; 222k text configuration):

Table with columns: Sampling, Original template + hermes, Qwen3.8 template (chat_template-xml-tools.jinja) + qwen3_coder
Sampling	Original template + `hermes`	Qwen3.8 template (`chat_template-xml-tools.jinja`) + `qwen3_coder`
`reasoning_effort` xhigh, no temperature (model default 1.0)	3/15	14/15
xhigh, temperature 0.3	5/15	15/15
reasoning disabled, temperature 0.3	13/15	15/15
`reasoning_effort` low, temperature 0.3	8/15	15/15

With hermes, the failed answers contained XML tool calls (15), malformed JSON (10) or nothing (6; only in the session whose history already contained empty answers). The one failure with the XML template was not a format error: in the multi-turn session with empty answers in the history, the model claimed a result without calling the tool. With the XML template, multi-step tasks run from opencode (list, count lines, read a file, write a summary; 4–5 tool calls with streaming) succeeded 3/3. An earlier short test (10 sysadmin tasks, 3 tools) gave 10/10 with hermes; it did not reveal the problem.

Reasoning length (6 problems with checkable answers — arithmetic, word problem, counting, logic, subnetting, systemd — × 3 tries, temperature 0.6, top_p 0.95; reasoning tokens counted with /tokenize; total over 18 requests, compared with the base Qwen3.8 quantized with the same recipe):

Table with columns: reasoning_effort, Qwen3.8-27B (same recipe), Qwopus3.8-27B-Flash-V2, Correct (Qwen3.8 / Qwopus)
`reasoning_effort`	Qwen3.8-27B (same recipe)	Qwopus3.8-27B-Flash-V2	Correct (Qwen3.8 / Qwopus)
low	7,304	8,549	18/18 · 18/18
medium	8,592	9,534	18/18 · 18/18
xhigh	8,605	7,306	17/18 · 18/18

On these short problems both models reason about the same amount (±15 %, within the run-to-run spread), and the level does not regulate it. No answer hit the 6,000-token limit.

With 8 tries per problem (48 requests per level, temperature 0.6): 23,524 / 28,624 / 21,145 reasoning tokens for low / medium / xhigh on 2026-09-30 (47 · 46 · 48 correct, one answer cut at 6,000 tokens), and 21,854 / 23,177 / 20,852 on 2026-10-02 (48 · 48 · 48). Agent tasks with opencode (4 sysadmin tasks × 2, xhigh, temperature 0.3, our setting until 2026-10-13): 7/8 with 4,958 reasoning tokens (2026-09-30), 6/8 with 2,741 (2026-10-02). See the Swift 1.5 card for the three-model comparison.

Quality: KL divergence against BF16 (2026-10-02, exllamav3's qbench, method in Family comparison): internal evaluation set KLD 0.0357 (median 0.0129), perplexity 1.7436 vs 1.6706 for BF16; public trace 0.0251 (median 0.0024), perplexity 1.4707 vs 1.4288.

Vision (MTP 3 INT8 + prefix caching, --limit-mm-per-prompt '{"image":1,"video":0}', --mm-processor-kwargs '{"max_pixels":1003520}', 6.5 GiB KV cache → 173,559 tokens, 160k context). Synthetic test images with known ground truth:

Table with columns: Test, Result
Test	Result
Terminal screenshot of a failed `systemctl status` (1280×800)	correct service, exact error line (`bind() ... :8443 failed (98: Address already in use)`)
Chart with numeric labels (1280×640)	3/3 values, correct derived percentage
Dense `nft list ruleset` with 38 rules (1120×896, ≈1 MP)	3/3: found the one `backup-agent` rule (port, source network) and the default policy
Prefix caching with an image (second turn)	2.1 s → 1.5 s, correct follow-up answer
≈1 MP image + 115k-token text prompt	correct, peak VRAM 23,596 MiB (980 MiB free)

Images with the new 4-image battery (2026-10-02, vision configuration: 176,128 context, KV 7.14e9 B, MTP 3, --limit-mm-per-prompt '{"image":1,"video":0}'): 4/4 tasks, 18/18 checks (failed-service screenshot 3/3, chart 3/3, dense firewall ruleset 3/3, port held by another process with a dead PID as a trap 6/6), follow-up turn with prefix cache OK, 957 MiB really free at the end of the run (not a long-prompt peak), no engine errors.

Quality sanity checks (small in-house battery, temperature 0 unless noted): arithmetic, a multi-step word problem (with reasoning), a Python function with asserts and multi-turn memory were all correct. A free-form historical summary (Spanish) contained several factual errors; a factual question in Catalan about a local institution was answered correctly. The same battery on the sibling Qwen3.8-27B checkpoint gave the opposite pattern (better history, wrong location), so no winner can be declared from these checks.

Validated serving configurations (run as systemd services, 24 GB RTX 3090)

Each configuration below was started as a real service and checked with a near-maximum-length prompt.

Table with columns: Configuration, Context (KV pool), Checks, Peak VRAM (free)
Configuration	Context (KV pool)	Checks	Peak VRAM (free)
Text, MTP 3 (INT8 head), prefix caching, 2 seqs, XML template + `qwen3_coder` (dedicated GPU)	222,000 (223,441)	8/8 facts at 205,792 tokens (347 s, 593 tok/s prefill); decode 97.9 / 92.1 tok/s (t=0 / 0.6), 174 / 171 tok/s with 2 requests; ≈2.9 accepted tokens per step; real opencode request 59/60	24,070 MiB (506 MiB)
Vision + MTP 3 (INT8 head), prefix caching, 2 seqs	176,128 (178,923)	3/3 test images; image + 127k-token prompt correct (validated with the original template; the XML template only changes tool formatting)	23,752 MiB (824 MiB)
Text, no MTP, 4 seqs (`main`)	262,144 (271,342)	8/8 facts at 246,276 tokens	23,816 MiB (760 MiB)

(The 222k row is the configuration before the two memory settings; with them the peak is 23,762 MiB, 361 MiB really free, see Important notes.)

Note: with MTP enabled, KV capacity grows in block-sized steps: always check the GPU KV cache size line after changing the KV bytes. For a GPU that also drives a display, use 8,350,000,000 bytes and --max-model-len 212800 (≈0.8 GB free at peak).

Alternative: DFlash 2 draft (tested, not recommended on one 24 GB GPU)

Instead of the MTP head, this checkpoint also works with the DFlash 2 draft z-lab/Qwen3.8-27B-DFlash2, quantized for 24 GB GPUs in JRamirez-UAB/Qwen3.8-27B-DFlash2-W4A16 (INT4 in main, INT8 in w8a16) — the draft was trained for the base Qwen3.8 but still accepts well on this fine-tune. Measured on one RTX 3090 (vLLM 0.29.0, 7 speculative tokens, fp8 KV cache, 2 sequences):

Table with columns: Speculation, Context (KV pool), Accepted tokens per step (decode · reasoning · tools), Decode, 1 request (t=0 / 0.6), 2 requests
Speculation	Context (KV pool)	Accepted tokens per step (decode · reasoning · tools)	Decode, 1 request (t=0 / 0.6)	2 requests
MTP 3 (INT8 head), this card's text configuration	222,000 (223,441)	≈2.9	98 / 92 tok/s	174 / 171 tok/s
DFlash 2 W4A16 draft, 6.3 GB KV cache	69,842	3.70 · 4.51 · 4.58	104 / 115 tok/s	169 / 165 tok/s
DFlash 2 W8A16 draft, 5.5 GB KV cache	60,854	3.75 · 4.78 · 4.97	108 / 101 tok/s	184 / 173 tok/s

The draft accepts more tokens per step, but on one GPU it gives only +10–25 % with one request and nothing with two, and its KV cache (≈90 KB per token against ≈35 KB with MTP in vLLM 0.29) limits the context to ≈60–70k tokens. The draft is meant for a two-GPU setup, where there is room for a long context; we do not have two GPUs, so that setup has not been tested. Details and commands on the draft's card.

Sampling and repetition loops

Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 3.85 build of Qwopus3.8-27B-Flash-V2 (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.

Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Loops / 30	3	1	2	1	2	0	0	0
Answers that finished before 16,000 tokens / 30	30	30	30	30	30	30	30	30

From temperature 0.8 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.

Other models, same test and setup (loops / 30):

Table with columns: Model (build), 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Model (build)	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Qwen3.8-27B · EXL3 3.85	12	12	7	8	1	1	1	0
Qwen3.8-27B-pi · EXL3 3.85	9	4	9	10	4	2	1	0
Swift 1.5 (Qwen3.8-27B) · EXL3 3.85	7	5	3	4	3	1	1	0
Ornith-1.5-9B · EXL3 6.00 (16 GB)	6	2	3	1	1	0	0	0
NeoHorse-1-9B · EXL3 6.00 (16 GB)	6	3	3	1	0	0	0	0
Qwen3.6-35B-A3B · EXL3 4.00	0	0	1	1	0	0	0	0
Ornith-1.5-35B-A3B · EXL3 4.00	1	0	0	0	0	0	0	0
Qwopus3.6-35B-A3B-Coder · EXL3 4.00	9	13	12	6	11	6	5	4
Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM	24	24	21	21	17	8	0	0
Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM	—	—	13	7	5	2	0	0

Requirements & install (vLLM with a small patch)

Stock vLLM (checked on 0.29.0 and 0.30.0) does not pass quant_config to the embedding layer of qwen3_5 models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). The 4-line patch in patches/qwen3_5-embed-quant.patch fixes it for the main model and for the MTP module. The patch is taken from syv-ai/qwen38-27b-rtx3090 (Apache-2.0).

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=auto        # torch 2.13.0VLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

First download the tool-calling chat template (it is passed to vLLM as a local file):

bash

hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB chat_template-xml-tools.jinja --revision mtp-int8 --local-dir .

With MTP (recommended; mtp-int8 revision), text only, 222k context, 2 concurrent sequences (for a GPU used only for the LLM; 361 MiB really free at peak with the two memory settings, see Important notes):

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --language-model-only \  --max-model-len 222000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8640000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

With MTP and vision (mtp-int8 revision), 172k context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 176128 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7140000000 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

Without MTP, full 262k context, 4 concurrent sequences:

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB \  --language-model-only --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 9101639680 --max-num-seqs 4 --enable-prefix-caching \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

8 concurrent sessions with MTP (text, shared 200k KV pool): the 222k text command with --max-model-len 200000 --kv-cache-memory-bytes 8150000000 --max-num-seqs 8 (validated, 751 MiB really free at peak; see Benchmarks). With 8 sequences vLLM 0.29 runs out of memory while capturing CUDA graphs at ≥222k.

For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 222,000 tokens, MTP 3. Aggregate decode 481 tok/s with 8 requests (Quick start: 190 with 8); a single request runs at 113 tok/s (114 with the Quick start); 8 of 8 simultaneous sessions returned all 8 needle facts; 174 MiB of VRAM really free. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). What was measured with opencode:

Tool calling with the real opencode request: the actual request that opencode sends (its long system prompt and its 11 tools), captured and replayed against vLLM, in three cases (fresh request, after a long story, full multi-turn session), several tries each.
End-to-end runs with opencode run: multi-step tasks (list files, count lines, read a file, write a summary) with real tool execution and streaming.

Context, speed and memory figures were measured directly against the vLLM API, not through opencode.

Start vLLM with --served-model-name qwopus3.8-27b (the model key in opencode must match it), then add a provider to ~/.config/opencode/config.json:

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwopus3.8-27b": {          "name": "Qwopus3.8-27B-Flash-V2 (local)",          "tool_call": true,          "temperature": true,          "limit": { "context": 222000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 },    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 }  }}

Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and in our agent tests (10 real sysadmin tasks × 3 runs with opencode run) 1.0 scored the same as 0.3 within one task: 29/30 at 1.0 and 30/30 at 0.3 for Qwen3.8-27B, -pi and Swift 1.5 27B; Qwopus3.8, served with its chat_template-xml-tools.jinja, 28/30 at 1.0 and 26/30 at 0.3.

Parameters that matter:

limit.context = vLLM's --max-model-len (222,000 in the text configuration above). opencode compacts the history at about context − output, so output: 22000 makes it compact at ≈200k tokens of input and never exceed the server's limit.
Temperature 0.3 for the agents, as recommended by the Qwopus author for agentic/coding work (the repository default is 1.0). With the real opencode request: 15/15 tool calls at 0.3, 14/15 at 1.0 (20 tries per case on 2026-10-02: 60/60 at 0.3, 57/60 at 1.0).
Declare "temperature": true in the model. Without it opencode does not send any temperature to a custom model, even if the agent sets one (checked by capturing the requests).
Reasoning levels: the variants pass chat_template_kwargs; reasoning_effort accepts only xhigh, medium and low (high returns HTTP 400). They are prompt instructions and do not limit the amount of reasoning (see the notes above).
Tool calling needs --chat-template chat_template-xml-tools.jinja and --tool-call-parser qwen3_coder (see Important notes). With the original template and hermes, opencode receives empty answers and the agent does nothing.
Start opencode inside a project directory, not in your home directory: before each step opencode takes a git snapshot of the directory, and on a large home directory that takes minutes and the request is never sent.
For opencode run from a script, close stdin (opencode run "..." </dev/null); otherwise it may wait for input and hang at start-up.

Important notes

Real VRAM headroom (corrected 2026-09-30): the driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use, and the "MiB free" figures in this card were computed as 24,576 − used, so they over-estimate the headroom by 453 MiB. Check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv. The original 222k command left only ≈50–70 MiB really free, and the same configuration crashed once with out-of-memory on the Swift 1.5 checkpoint (same recipe and size) during an agent request. The 222k command above now adds --max-num-batched-tokens 1024 and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: measured on this checkpoint the peak drops to 23,762 MiB (361 MiB really free) with the same prefill (586 tok/s at 205,792 tokens, 8/8 facts) and decode speed, no OOM. (The "really free" values in the 8-session rows of Benchmarks were read directly with nvidia-smi.)
For tool calling, use the base Qwen3.8 template with --tool-call-parser qwen3_coder (--chat-template chat_template-xml-tools.jinja). The original Qwopus V2 template asks for JSON tool calls (Hermes style), but the model often answers in Qwen3.8's XML format (<function=...><parameter=...>) or with malformed JSON, and the hermes parser silently drops those calls: the client receives an empty answer with finish_reason=stop. The two templates differ only in how tools and tool calls are rendered (reasoning, vision and history are identical). An earlier version of this card recommended hermes; that was based on a short 3-tool test that did not reproduce real agent use.
Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization. With this hybrid (GatedDeltaNet) model, vLLM 0.29's memory profiling under-estimates the prefill activations (it measured 0.4–0.7 GiB; the real peak is ≈2 GiB). With a percentage it over-allocates the KV cache and runs out of memory on the first long prompt. The byte values above were validated with a near-maximum-length prompt; they leave ≈0.8–1.0 GB of VRAM free.
On Ampere (sm86), fp8 KV cache only works with the FlashInfer attention backend (the default). The Triton backend refuses fp8 KV below sm89.
reasoning_effort (via chat_template_kwargs) accepts only xhigh (default), medium and low; other values return HTTP 400. enable_thinking: false disables reasoning. The levels are prompt instructions, not token budgets, and in our measurements they do not regulate the amount of reasoning (see Benchmarks). For a hard cap, send thinking_token_budget in the request: with --reasoning-parser qwen3, vLLM 0.29 enforces it (also with MTP) by forcing </think> when the budget is reached.
Two fields of the config.json generated by llm-compressor had to be adjusted for vLLM (already applied here): the lm_head target is the regex re:.*lm_head$, and the MTP ignore rule is re:^mtp\.(fc|layers\..*)$, so that the MTP shares the quantized embeddings and lm_head.

Quantization recipe

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 128, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT4 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`, `conv1d`	kept in BF16 (small and sensitive)
MTP head (`mtp.*`)	BF16 in `main`; INT8 g128 RTN in `mtp-int8` (shares the INT4 embeddings / `lm_head` of the main model)
Vision tower	kept in BF16 (not needed for text; use `--language-model-only`)

Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens from HuggingFaceH4/ultrachat_200k (the first 512 of the train_sft split, shuffled with seed 42; chat template applied). Only the GPTQ linear layers use it: embed_tokens / lm_head (INT4) and the INT8 MTP head of the mtp-int8 revision are round-to-nearest, with no data.
Quantization took 26.7 min on a single RTX PRO 6000 Blackwell (96 GB).
Tokenizer and chat template are byte-identical to the base model. For tool calling we recommend the extra template chat_template-xml-tools.jinja (see Important notes).
Format: compressed-tensors pack-quantized (Marlin kernels in vLLM).

Integrity and validation

What was checked for this checkpoint, how, and where the numbers are (measured on 2026-10-14; the method and every figure are in Family comparison).

Artifact integrity

Every weight file is checked against our local copy with SHA-256 before publication (SHA256SUMS in this repo).
The measurements used the weights downloaded from this repo at commit 1a992503d5e7; later commits change only metadata, documentation or the plugin, never the .safetensors files.
vLLM loads every tensor listed in model.safetensors.index.json on start-up (a missing tensor or a wrong shape would stop it): checked from a fresh download.

Quantization

Quantization recipe and calibration in recipe.yaml and scripts/ (llm-compressor GPTQ, compressed-tensors format).
Fidelity: KL divergence against the BF16 Qwopus3.8-27B-Flash-V2 with exllamav3's qbench on two traces (ours and turboderp's public one).

Chat template

The published template renders correctly in 40 edge cases (empty and very long system prompt, no tools, two tool calls in one message, nested JSON and multi-line arguments, images with tools, two images, earlier reasoning, multi-turn with several tool results) × the reasoning settings (enable_thinking: false, reasoning_effort low / xhigh): all pass (prueba_plantillas.py). With Transformers directly, pass tool-call arguments as dicts (vLLM converts JSON strings itself).

Runtime

Load, long-context needle, vision, tool calls with the real opencode request, 10 agent tasks × 3 runs, concurrency and VRAM: see Family comparison; engine errors during the run: 0.
On a desktop RTX 3090 that also drives the display (300 W): Quick start at 222,000 tokens: 301 MiB really free; vision config at 176,128 tokens: 921 MiB really free.

Scope: results apply to the revision, software stack (vLLM 0.31.0 + the embedding patch), hardware and settings stated above; other setups are not tested.

Limitations

Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with sudo, and give the agent a non-privileged account and a test machine where you can.
Quality has not been evaluated with standard benchmarks: only KL divergence against BF16, long-context retrieval, tool calling, vision, reasoning problems, agent tasks and a small sanity battery (above) were run on this checkpoint. Calibration used general chat data (ultrachat).
The INT4 RTN embeddings / lm_head are the main source of the KL divergence (KLD 0.036; our EXL3 3.85 bpw version of this model measures 0.0060).
The fp8 KV cache uses the default scale 1.0 (no calibrated KV scales). Long-context retrieval was nevertheless 8/8 up to 246k tokens.
All numbers come from one machine and one software stack; expect different figures on other drivers, power limits or vLLM versions.
Vision was validated with synthetic test images only (terminal screenshot, chart, dense text); not with natural photos or video.

Feedback welcome

I would be grateful for reports of installations of this checkpoint, both working and failed ones. Please open a discussion in the Community tab with your GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error message). Failed attempts are just as useful as successful ones: they help to document which setups work.

Credits & license

Credits & acknowledgements

This work is directly inspired by Marcin116 and their checkpoint Qwen3.8-27B-W4A16-AutoRound-fast-embed-int4, which showed that Qwen3.8-27B can serve very long contexts on RTX 3090-class GPUs by quantizing the token embeddings to INT4. We served and measured that checkpoint on our RTX 3090, and then set out to apply the same idea to the Qwopus V2 fine-tune.

What we took from that line of work:

the INT4 embed_tokens / lm_head layout (group 128, compressed-tensors pack-quantized), which is what makes the 262k context fit;
keeping the GatedDeltaNet in_proj_a / in_proj_b projections in BF16;
the vLLM patch that routes quantized embeddings through qwen3_5 and its MTP module.

Marcin116's checkpoint itself builds on born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast and on the syvai/qwen3.8-27b-3090-fast-variant companion package (patch: syv-ai/qwen38-27b-rtx3090), developed with the Club3090 community. Many thanks to all of them. Any mistakes in this checkpoint are ours.

What is different here: a different base model (Qwopus3.8-27B-Flash-V2), GPTQ (llm-compressor) instead of AutoRound for the linear layers, the MTP head kept in BF16 or INT8 instead of INT4, and a full set of measurements on a single RTX 3090 (above).

Base model: Jackrong/Qwopus3.8-27B-Flash-V2, built on Qwen/Qwen3.8-27B.
INT4-embedding layout: Marcin116 / born2bewild; vLLM patch: syv-ai.
Tools: llm-compressor, compressed-tensors, vLLM.
KL-divergence evaluation: qbench from exllamav3 (turboderp) and its public evaluation trace.

License

Apache-2.0, same as the base model. The included patch is Apache-2.0 (syv-ai). This is a community quantization, not an official release of Jackrong or Qwen.

What is in this repository

Which one to use

All Qwopus3.8-27B-Flash-V2 variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):

Table with columns: Variant, Weights, Context on one RTX 3090, KLD vs BF16, Agent tasks, Decode, 1 request, Pick it for
Variant	Weights	Context on one RTX 3090	KLD vs BF16	Agent tasks	Decode, 1 request	Pick it for
GPTQ INT4 (this repo)	14.21 GiB	222,000 (text) · 176,128 with vision	0.0357	29/30	114.3 tok/s	the longest context on stock vLLM (+ a 4-line patch): 222,000 text-only, or 176,128 with vision
GPTQ INT8 g64	15.75 GiB	176,000 (text) · 131,072 with vision	0.0145	25/30	97.3 tok/s	a more faithful GPTQ (KLD 0.0145) on stock vLLM (+ the same patch), with less context
EXL3 3.85	13.62 GiB	215,000 with vision	0.0060	27/30	86.9 tok/s	the closest to BF16 (KLD 0.0060) with vision at 215,000; needs exllamav3 + our plugin
EXL3 3.70	13.19 GiB	226,000 with vision	0.0073	25/30	83.9 tok/s	more context (226,000) and 4 sessions with vision, a little less faithful than 3.85

Variants / revisions

Table with columns: Revision, MTP head, Size, Notes
Revision	MTP head	Size	Notes
`main`	BF16 (0.79 GiB)	14.6 GiB	reference; use without speculation for the full 262k context
`mtp-int8`	INT8 g128 (0.40 GiB)	14.21 GiB	recommended with MTP: same acceptance as BF16, +≈0.4 GiB for KV cache

Files and reproducibility

scripts/quantize.py: exact quantization script (llm-compressor); recipe.yaml: the recipe as serialized by llm-compressor.
scripts/mtp_quant.py: script that builds the mtp-int8 revision from main.
config.json.orig-llmcompressor: config as emitted by llm-compressor, before the two vLLM fixes described above.
chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
patches/: vLLM patch and its Apache-2.0 license.
SHA256SUMS: checksums of the model files.

Results

Family comparison (Qwopus3.8-27B-Flash-V2, same conditions)

Table with columns: Variant, Weights, KLD mean / median (internal · public), Perplexity (BF16 1.6706 · pub. 1.4288), Needle (8 facts) near max. context, Tools: real opencode request, Agent tasks (10 × 3), Reasoning xhigh: correct · tokens, Vision
Variant	Weights	KLD mean / median (internal · public)	Perplexity (BF16 1.6706 · pub. 1.4288)	Needle (8 facts) near max. context	Tools: real opencode request	Agent tasks (10 × 3)	Reasoning `xhigh`: correct · tokens	Vision
GPTQ W4A16 + INT4 embeddings (`mtp-int8`, this repo)	14.21 GiB	0.0357 / 0.0129 · pub. 0.0251 / 0.0024	1.7436 · pub. 1.4707	8/8 at 204,551	55/60	29/30	47/48 · 18,090	15/15
GPTQ W4A16 g64 + INT8 embeddings (`main`)	15.75 GiB	0.0145 / 0.0047 · pub. 0.0115 / 0.0011	1.7029 · pub. 1.4507	8/8 at 162,208	57/60	25/30	48/48 · 17,965	15/15
EXL3 3.85 bpw (`main`)	13.62 GiB	0.0060 / 0.0018 · pub. 0.0048 / 0.0004	1.6815 · pub. 1.4379	8/8 at 197,991	54/60	27/30	48/48 · 23,460	15/15
EXL3 3.70 bpw (`3.70bpw`)	13.19 GiB	0.0073 / 0.0022 · pub. 0.0060 / 0.0005	1.6856 · pub. 1.4409	8/8 at 208,158	57/60	25/30	48/48 · 26,996	15/15

Table with columns: Variant, Context on one RTX 3090, Sessions, Prefill near max. context, Decode, 1 / 2 / 8 requests (tok/s), MTP tokens per step, VRAM really free at peak
Variant	Context on one RTX 3090	Sessions	Prefill near max. context	Decode, 1 / 2 / 8 requests (tok/s)	MTP tokens per step	VRAM really free at peak
GPTQ INT4	222,000 (text) · 176,128 with vision	2	603 tok/s	114.3 / 192.2 / 189.5	3.20	284 MiB · vision config: 930 MiB
GPTQ INT8 g64	176,000 (text) · 131,072 with vision	2	644 tok/s	97.3 / 173.0 / 175.1	3.21	196 MiB · vision config: 840 MiB
EXL3 3.85	215,000 with vision	2	554 tok/s	86.9 / 140.5 / 145.3	3.40	320 MiB
EXL3 3.70	226,000 with vision	4	543 tok/s	83.9 / 145.7 / 258.8	3.21	148 MiB

How to read it:

KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign.
Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test.
Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run.
VRAM really free: lowest memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB.
Older sections below keep their own date and setup; use this table to compare variants.

Table with columns: Variant, KV pool (tokens), Max sessions, 32k each, 64k each, 128k each, Full context each
Variant	KV pool (tokens)	Max sessions	32k each	64k each	128k each	Full context each
GPTQ INT4	223,441	2	2	2	1	1 × 222,000
GPTQ INT4 (vision config)	178,923	2	2	2	1	1 × 176,128
GPTQ INT8 g64	178,816	2	2	2	1	1 × 176,000
GPTQ INT8 g64 (vision config)	132,423	2	2	2	1	1 × 131,072
EXL3 3.85	216,433	2	2	2	1	1 × 215,000
EXL3 3.70	231,757	4	4	3	1	1 × 226,000

Table with columns: Variant, Configuration, Decode by simultaneous requests (aggregate tok/s), N sessions at once, needle 8/8, Tools (5 × 3), VRAM really free
Variant	Configuration	Decode by simultaneous requests (aggregate tok/s)	N sessions at once, needle 8/8	Tools (5 × 3)	VRAM really free
GPTQ INT4	Quick start (2 sessions, MTP 3)	1: 114 · 2: 192 · 8: 190	—	55/60	284 MiB
GPTQ INT4	many users (8 sessions of up to 222,000 tokens, MTP 3)	1: 113 · 2: 190 · 4: 340 · 8: 481	8/8 of 22k	15/15	174 MiB
GPTQ INT8 g64	Quick start (2 sessions, MTP 3)	1: 97 · 2: 173 · 8: 175	—	57/60	196 MiB
GPTQ INT8 g64	many users (8 sessions of up to 165,000 tokens, MTP 3)	1: 96 · 2: 166 · 4: 316 · 8: 460	8/8 of 22k	14/15	386 MiB
EXL3 3.85	Quick start (2 sessions, MTP 3)	1: 87 · 2: 140 · 8: 145	—	54/60	320 MiB
EXL3 3.85	many users (8 sessions of up to 215,000 tokens, MTP 3)	1: 88 · 2: 142 · 4: 257 · 8: 314	8/8 of 22k	14/15	138 MiB
EXL3 3.70	Quick start (4 sessions, MTP 3)	1: 84 · 2: 146 · 8: 259	—	57/60	148 MiB
EXL3 3.70	many users (8 sessions of up to 220,000 tokens, MTP 3)	1: 78 · 2: 143 · 4: 254 · 8: 293	8/8 of 22k	15/15	358 MiB

Benchmarks and validated configurations (single RTX 3090 24 GB)

Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.

Long context: 8 distinct facts spread from 3 % to 97 % depth, all 8 must be returned (greedy)

Table with columns: Configuration, Prompt tokens, Recovered, Time, Prefill
Configuration	Prompt tokens	Recovered	Time	Prefill
No MTP, 262k	57,874	8/8	59.8 s	968 tok/s
No MTP, 262k	246,276	8/8	435.2 s	566 tok/s
MTP 1 token (BF16 head), 208k	199,086	8/8	326.5 s	610 tok/s
MTP 3 tokens (INT8 head), 200k, prefix caching	189,270	8/8	306.4 s	618 tok/s

Decode speed and MTP (Qwopus MTP head)

Table with columns: Configuration, Context, 1 request, t=0 / t=0.6, 4 concurrent, aggregate, Accepted tokens per step, Per-position acceptance
Configuration	Context	1 request, t=0 / t=0.6	4 concurrent, aggregate	Accepted tokens per step	Per-position acceptance
No speculation	262k	49.9 / 50.5 tok/s	177 tok/s	—	—
MTP, 1 token	208k	76.5 / 72.6 tok/s	287 tok/s	1.85	85 %
MTP, 2 tokens	208k	88.3 / 84.8 tok/s	304 tok/s	2.43	82 / 62 %
MTP, 3 tokens (BF16 head)	200k	95.5 / 89.9 tok/s	313 tok/s	≈2.98	83 / 64 / 50 %
MTP, 3 tokens (INT8 head)	200k	95.5 / 95.8 tok/s	168 tok/s (2 seqs)	≈2.91	83 / 63 / 46 %
MTP, 3 tokens (INT4 head, not published)	200k	94.8 / 92.3 tok/s	174 tok/s (2 seqs)	2.87	83 / 61 / 43 %

Speculative decoding with MTP does not change the output: every drafted token is verified by the main model. Prefill cost of MTP: −1 to −3 %.

Prefix caching in an agentic session (≈90k-token history, each turn appends ≈2k tokens of tool output; time to first token)

Table with columns: Turn, 0, 1, 2, 3, 4, 5, 6
Turn	0	1	2	3	4	5	6
Without prefix caching	98.5 s	103.5 s	108.0 s	112.8 s	117.4 s	122.1 s	126.9 s
With `--enable-prefix-caching`	100.5 s	5.7 s	5.7 s	5.6 s	5.4 s	5.2 s	7.5 s

KV capacity cost of prefix caching: −2 % (270,663 → 265,356 tokens). With MTP (INT8) enabled as well, later turns take ≈6.8 s.

Concurrency (4 × 58k-token prompts at once, no MTP): 8/8 retrieval in all four, 988 tok/s aggregate prefill (same efficiency as one request).

VRAM (peak, 24,576 MiB total): 23,816 MiB without MTP (262k, 4 seqs); 23,592 MiB with MTP-INT8 (200k, 2 seqs, prefix caching).

Table with columns: Configuration, Turn: time to first token / total (median), First turn (no cache), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0 / 0.6), Really free (min)
Configuration	Turn: time to first token / total (median)	First turn (no cache)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0 / 0.6)	Really free (min)
150,000, KV 6.25e9	2.6 / 3.3 s	11.6 s	8 × 16.5k, 8/8 in all, 1,129 tok/s	462.8 / 447.3 tok/s	2,571 MiB
200,000, KV 8.15e9	2.6 / 3.3 s	11.6 s	8 × 22k, 8/8 in all, 1,105 tok/s	459.5 / 432.7 tok/s	751 MiB

Table with columns: Sampling, Original template + hermes, Qwen3.8 template (chat_template-xml-tools.jinja) + qwen3_coder
Sampling	Original template + `hermes`	Qwen3.8 template (`chat_template-xml-tools.jinja`) + `qwen3_coder`
`reasoning_effort` xhigh, no temperature (model default 1.0)	3/15	14/15
xhigh, temperature 0.3	5/15	15/15
reasoning disabled, temperature 0.3	13/15	15/15
`reasoning_effort` low, temperature 0.3	8/15	15/15

Table with columns: reasoning_effort, Qwen3.8-27B (same recipe), Qwopus3.8-27B-Flash-V2, Correct (Qwen3.8 / Qwopus)
`reasoning_effort`	Qwen3.8-27B (same recipe)	Qwopus3.8-27B-Flash-V2	Correct (Qwen3.8 / Qwopus)
low	7,304	8,549	18/18 · 18/18
medium	8,592	9,534	18/18 · 18/18
xhigh	8,605	7,306	17/18 · 18/18

On these short problems both models reason about the same amount (±15 %, within the run-to-run spread), and the level does not regulate it. No answer hit the 6,000-token limit.

Table with columns: Test, Result
Test	Result
Terminal screenshot of a failed `systemctl status` (1280×800)	correct service, exact error line (`bind() ... :8443 failed (98: Address already in use)`)
Chart with numeric labels (1280×640)	3/3 values, correct derived percentage
Dense `nft list ruleset` with 38 rules (1120×896, ≈1 MP)	3/3: found the one `backup-agent` rule (port, source network) and the default policy
Prefix caching with an image (second turn)	2.1 s → 1.5 s, correct follow-up answer
≈1 MP image + 115k-token text prompt	correct, peak VRAM 23,596 MiB (980 MiB free)

Validated serving configurations (run as systemd services, 24 GB RTX 3090)

Each configuration below was started as a real service and checked with a near-maximum-length prompt.

Table with columns: Configuration, Context (KV pool), Checks, Peak VRAM (free)
Configuration	Context (KV pool)	Checks	Peak VRAM (free)
Text, MTP 3 (INT8 head), prefix caching, 2 seqs, XML template + `qwen3_coder` (dedicated GPU)	222,000 (223,441)	8/8 facts at 205,792 tokens (347 s, 593 tok/s prefill); decode 97.9 / 92.1 tok/s (t=0 / 0.6), 174 / 171 tok/s with 2 requests; ≈2.9 accepted tokens per step; real opencode request 59/60	24,070 MiB (506 MiB)
Vision + MTP 3 (INT8 head), prefix caching, 2 seqs	176,128 (178,923)	3/3 test images; image + 127k-token prompt correct (validated with the original template; the XML template only changes tool formatting)	23,752 MiB (824 MiB)
Text, no MTP, 4 seqs (`main`)	262,144 (271,342)	8/8 facts at 246,276 tokens	23,816 MiB (760 MiB)

(The 222k row is the configuration before the two memory settings; with them the peak is 23,762 MiB, 361 MiB really free, see Important notes.)

Alternative: DFlash 2 draft (tested, not recommended on one 24 GB GPU)

Table with columns: Speculation, Context (KV pool), Accepted tokens per step (decode · reasoning · tools), Decode, 1 request (t=0 / 0.6), 2 requests
Speculation	Context (KV pool)	Accepted tokens per step (decode · reasoning · tools)	Decode, 1 request (t=0 / 0.6)	2 requests
MTP 3 (INT8 head), this card's text configuration	222,000 (223,441)	≈2.9	98 / 92 tok/s	174 / 171 tok/s
DFlash 2 W4A16 draft, 6.3 GB KV cache	69,842	3.70 · 4.51 · 4.58	104 / 115 tok/s	169 / 165 tok/s
DFlash 2 W8A16 draft, 5.5 GB KV cache	60,854	3.75 · 4.78 · 4.97	108 / 101 tok/s	184 / 173 tok/s

Sampling and repetition loops

Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Loops / 30	3	1	2	1	2	0	0	0
Answers that finished before 16,000 tokens / 30	30	30	30	30	30	30	30	30

From temperature 0.8 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.

Other models, same test and setup (loops / 30):

Table with columns: Model (build), 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Model (build)	0.3	0.4	0.5	0.6	0.7	0.8	0.9	1.0
Qwen3.8-27B · EXL3 3.85	12	12	7	8	1	1	1	0
Qwen3.8-27B-pi · EXL3 3.85	9	4	9	10	4	2	1	0
Swift 1.5 (Qwen3.8-27B) · EXL3 3.85	7	5	3	4	3	1	1	0
Ornith-1.5-9B · EXL3 6.00 (16 GB)	6	2	3	1	1	0	0	0
NeoHorse-1-9B · EXL3 6.00 (16 GB)	6	3	3	1	0	0	0	0
Qwen3.6-35B-A3B · EXL3 4.00	0	0	1	1	0	0	0	0
Ornith-1.5-35B-A3B · EXL3 4.00	1	0	0	0	0	0	0	0
Qwopus3.6-35B-A3B-Coder · EXL3 4.00	9	13	12	6	11	6	5	4
Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM	24	24	21	21	17	8	0	0
Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM	—	—	13	7	5	2	0	0

Requirements & install (vLLM with a small patch)

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=auto        # torch 2.13.0VLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=auto        # torch 2.13.0VLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

First download the tool-calling chat template (it is passed to vLLM as a local file):

bash

hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB chat_template-xml-tools.jinja --revision mtp-int8 --local-dir .

hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB chat_template-xml-tools.jinja --revision mtp-int8 --local-dir .

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --language-model-only \  --max-model-len 222000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8640000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --language-model-only \  --max-model-len 222000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8640000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

With MTP and vision (mtp-int8 revision), 172k context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 176128 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7140000000 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 176128 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7140000000 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

Without MTP, full 262k context, 4 concurrent sequences:

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB \  --language-model-only --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 9101639680 --max-num-seqs 4 --enable-prefix-caching \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB \  --language-model-only --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 9101639680 --max-num-seqs 4 --enable-prefix-caching \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). What was measured with opencode:

Tool calling with the real opencode request: the actual request that opencode sends (its long system prompt and its 11 tools), captured and replayed against vLLM, in three cases (fresh request, after a long story, full multi-turn session), several tries each.
End-to-end runs with opencode run: multi-step tasks (list files, count lines, read a file, write a summary) with real tool execution and streaming.

Context, speed and memory figures were measured directly against the vLLM API, not through opencode.

Start vLLM with --served-model-name qwopus3.8-27b (the model key in opencode must match it), then add a provider to ~/.config/opencode/config.json:

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwopus3.8-27b": {          "name": "Qwopus3.8-27B-Flash-V2 (local)",          "tool_call": true,          "temperature": true,          "limit": { "context": 222000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 },    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 }  }}

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwopus3.8-27b": {          "name": "Qwopus3.8-27B-Flash-V2 (local)",          "tool_call": true,          "temperature": true,          "limit": { "context": 222000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 },    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 1.0 }  }}

Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and in our agent tests (10 real sysadmin tasks × 3 runs with opencode run) 1.0 scored the same as 0.3 within one task: 29/30 at 1.0 and 30/30 at 0.3 for Qwen3.8-27B, -pi and Swift 1.5 27B; Qwopus3.8, served with its chat_template-xml-tools.jinja, 28/30 at 1.0 and 26/30 at 0.3.

Parameters that matter:

limit.context = vLLM's --max-model-len (222,000 in the text configuration above). opencode compacts the history at about context − output, so output: 22000 makes it compact at ≈200k tokens of input and never exceed the server's limit.
Temperature 0.3 for the agents, as recommended by the Qwopus author for agentic/coding work (the repository default is 1.0). With the real opencode request: 15/15 tool calls at 0.3, 14/15 at 1.0 (20 tries per case on 2026-10-02: 60/60 at 0.3, 57/60 at 1.0).
Declare "temperature": true in the model. Without it opencode does not send any temperature to a custom model, even if the agent sets one (checked by capturing the requests).
Reasoning levels: the variants pass chat_template_kwargs; reasoning_effort accepts only xhigh, medium and low (high returns HTTP 400). They are prompt instructions and do not limit the amount of reasoning (see the notes above).
Tool calling needs --chat-template chat_template-xml-tools.jinja and --tool-call-parser qwen3_coder (see Important notes). With the original template and hermes, opencode receives empty answers and the agent does nothing.
Start opencode inside a project directory, not in your home directory: before each step opencode takes a git snapshot of the directory, and on a large home directory that takes minutes and the request is never sent.
For opencode run from a script, close stdin (opencode run "..." </dev/null); otherwise it may wait for input and hang at start-up.

Important notes

Real VRAM headroom (corrected 2026-09-30): the driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use, and the "MiB free" figures in this card were computed as 24,576 − used, so they over-estimate the headroom by 453 MiB. Check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv. The original 222k command left only ≈50–70 MiB really free, and the same configuration crashed once with out-of-memory on the Swift 1.5 checkpoint (same recipe and size) during an agent request. The 222k command above now adds --max-num-batched-tokens 1024 and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: measured on this checkpoint the peak drops to 23,762 MiB (361 MiB really free) with the same prefill (586 tok/s at 205,792 tokens, 8/8 facts) and decode speed, no OOM. (The "really free" values in the 8-session rows of Benchmarks were read directly with nvidia-smi.)
For tool calling, use the base Qwen3.8 template with --tool-call-parser qwen3_coder (--chat-template chat_template-xml-tools.jinja). The original Qwopus V2 template asks for JSON tool calls (Hermes style), but the model often answers in Qwen3.8's XML format (<function=...><parameter=...>) or with malformed JSON, and the hermes parser silently drops those calls: the client receives an empty answer with finish_reason=stop. The two templates differ only in how tools and tool calls are rendered (reasoning, vision and history are identical). An earlier version of this card recommended hermes; that was based on a short 3-tool test that did not reproduce real agent use.
Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization. With this hybrid (GatedDeltaNet) model, vLLM 0.29's memory profiling under-estimates the prefill activations (it measured 0.4–0.7 GiB; the real peak is ≈2 GiB). With a percentage it over-allocates the KV cache and runs out of memory on the first long prompt. The byte values above were validated with a near-maximum-length prompt; they leave ≈0.8–1.0 GB of VRAM free.
On Ampere (sm86), fp8 KV cache only works with the FlashInfer attention backend (the default). The Triton backend refuses fp8 KV below sm89.
reasoning_effort (via chat_template_kwargs) accepts only xhigh (default), medium and low; other values return HTTP 400. enable_thinking: false disables reasoning. The levels are prompt instructions, not token budgets, and in our measurements they do not regulate the amount of reasoning (see Benchmarks). For a hard cap, send thinking_token_budget in the request: with --reasoning-parser qwen3, vLLM 0.29 enforces it (also with MTP) by forcing </think> when the budget is reached.
Two fields of the config.json generated by llm-compressor had to be adjusted for vLLM (already applied here): the lm_head target is the regex re:.*lm_head$, and the MTP ignore rule is re:^mtp\.(fc|layers\..*)$, so that the MTP shares the quantized embeddings and lm_head.

Quantization recipe

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 128, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT4 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`, `conv1d`	kept in BF16 (small and sensitive)
MTP head (`mtp.*`)	BF16 in `main`; INT8 g128 RTN in `mtp-int8` (shares the INT4 embeddings / `lm_head` of the main model)
Vision tower	kept in BF16 (not needed for text; use `--language-model-only`)

Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens from HuggingFaceH4/ultrachat_200k (the first 512 of the train_sft split, shuffled with seed 42; chat template applied). Only the GPTQ linear layers use it: embed_tokens / lm_head (INT4) and the INT8 MTP head of the mtp-int8 revision are round-to-nearest, with no data.
Quantization took 26.7 min on a single RTX PRO 6000 Blackwell (96 GB).
Tokenizer and chat template are byte-identical to the base model. For tool calling we recommend the extra template chat_template-xml-tools.jinja (see Important notes).
Format: compressed-tensors pack-quantized (Marlin kernels in vLLM).

Integrity and validation

What was checked for this checkpoint, how, and where the numbers are (measured on 2026-10-14; the method and every figure are in Family comparison).

Artifact integrity

Every weight file is checked against our local copy with SHA-256 before publication (SHA256SUMS in this repo).
The measurements used the weights downloaded from this repo at commit 1a992503d5e7; later commits change only metadata, documentation or the plugin, never the .safetensors files.
vLLM loads every tensor listed in model.safetensors.index.json on start-up (a missing tensor or a wrong shape would stop it): checked from a fresh download.

Quantization

Quantization recipe and calibration in recipe.yaml and scripts/ (llm-compressor GPTQ, compressed-tensors format).
Fidelity: KL divergence against the BF16 Qwopus3.8-27B-Flash-V2 with exllamav3's qbench on two traces (ours and turboderp's public one).

Chat template

The published template renders correctly in 40 edge cases (empty and very long system prompt, no tools, two tool calls in one message, nested JSON and multi-line arguments, images with tools, two images, earlier reasoning, multi-turn with several tool results) × the reasoning settings (enable_thinking: false, reasoning_effort low / xhigh): all pass (prueba_plantillas.py). With Transformers directly, pass tool-call arguments as dicts (vLLM converts JSON strings itself).

Runtime

Load, long-context needle, vision, tool calls with the real opencode request, 10 agent tasks × 3 runs, concurrency and VRAM: see Family comparison; engine errors during the run: 0.
On a desktop RTX 3090 that also drives the display (300 W): Quick start at 222,000 tokens: 301 MiB really free; vision config at 176,128 tokens: 921 MiB really free.

Scope: results apply to the revision, software stack (vLLM 0.31.0 + the embedding patch), hardware and settings stated above; other setups are not tested.

Limitations

Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with sudo, and give the agent a non-privileged account and a test machine where you can.
Quality has not been evaluated with standard benchmarks: only KL divergence against BF16, long-context retrieval, tool calling, vision, reasoning problems, agent tasks and a small sanity battery (above) were run on this checkpoint. Calibration used general chat data (ultrachat).
The INT4 RTN embeddings / lm_head are the main source of the KL divergence (KLD 0.036; our EXL3 3.85 bpw version of this model measures 0.0060).
The fp8 KV cache uses the default scale 1.0 (no calibrated KV scales). Long-context retrieval was nevertheless 8/8 up to 246k tokens.
All numbers come from one machine and one software stack; expect different figures on other drivers, power limits or vLLM versions.
Vision was validated with synthetic test images only (terminal screenshot, chart, dense text); not with natural photos or video.

Feedback welcome

Credits & license

Credits & acknowledgements

What we took from that line of work:

the INT4 embed_tokens / lm_head layout (group 128, compressed-tensors pack-quantized), which is what makes the 262k context fit;
keeping the GatedDeltaNet in_proj_a / in_proj_b projections in BF16;
the vLLM patch that routes quantized embeddings through qwen3_5 and its MTP module.

Base model: Jackrong/Qwopus3.8-27B-Flash-V2, built on Qwen/Qwen3.8-27B.
INT4-embedding layout: Marcin116 / born2bewild; vLLM patch: syv-ai.
Tools: llm-compressor, compressed-tensors, vLLM.
KL-divergence evaluation: qbench from exllamav3 (turboderp) and its public evaluation trace.

License

Apache-2.0, same as the base model. The included patch is Apache-2.0 (syv-ai). This is a community quantization, not an official release of Jackrong or Qwen.

Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB

README

What is in this repository

Which one to use

Variants / revisions

Files and reproducibility

Results

Family comparison (Qwopus3.8-27B-Flash-V2, same conditions)

Benchmarks and validated configurations (single RTX 3090 24 GB)

Validated serving configurations (run as systemd services, 24 GB RTX 3090)

Alternative: DFlash 2 draft (tested, not recommended on one 24 GB GPU)

Sampling and repetition loops

Requirements & install (vLLM with a small patch)

Quick start (single RTX 3090, 24 GB)

Using it with opencode (how we tested)

Important notes

Quantization recipe

Integrity and validation

Limitations

Feedback welcome

Credits & license

Credits & acknowledgements

License

Explore FriendliAI today

README

What is in this repository

Which one to use

Variants / revisions

Files and reproducibility

Results

Family comparison (Qwopus3.8-27B-Flash-V2, same conditions)

Benchmarks and validated configurations (single RTX 3090 24 GB)

Validated serving configurations (run as systemd services, 24 GB RTX 3090)

Alternative: DFlash 2 draft (tested, not recommended on one 24 GB GPU)

Sampling and repetition loops

Requirements & install (vLLM with a small patch)

Quick start (single RTX 3090, 24 GB)

Using it with opencode (how we tested)

Important notes

Quantization recipe

Integrity and validation

Limitations

Feedback welcome

Credits & license

Credits & acknowledgements

License