JRamirez-UAB

Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB

README

License: apache-2.0

Family comparison (Qwopus3.8-27B-Flash-V2, single RTX 3090 24 GB)

The same columns are used in the cards of our other families (Qwen3.8-27B, Swift 1.5). For Qwopus only the 3.85 bpw EXL3 version is published (a 3.70 bpw one fell into repetition loops in agent tasks; see its card).

Table with columns: Model, Size, Context in 24 GB, KLD mean / median, Perplexity (BF16: 1.6706 · pub. 1.4288), Needle · tools at T=1.0 · T=0.3 · agent, Reasoning xhigh (tokens), Prefill, Decode with MTP, 1 / 2 sessions, Tokens per step, Accepted drafts (of 3)
Model	Size	Context in 24 GB	KLD mean / median	Perplexity (BF16: 1.6706 · pub. 1.4288)	Needle · tools at T=1.0 · T=0.3 · agent	Reasoning xhigh (tokens)	Prefill	Decode with MTP, 1 / 2 sessions	Tokens per step	Accepted drafts (of 3)
W4A16 GPTQ + INT4 embeddings (`mtp-int8`, this repo)	14.2 GiB	222,000 (text) · 176,128 (vision)	0.0357 / 0.0129 · pub. 0.0251 / 0.0024	1.7436 · pub. 1.4707	8/8 at 205,792 · 57/60 · 60/60 · 6/8	20,852	586 tok/s	97.9 / 174 tok/s	3.03–3.07	2.03–2.07
EXL3 3.85 bpw (`main`)	13.64 GiB

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 Qwopus3.8-27B-Flash-V2, measured with exllamav3's qbench. The first figure uses an internal sysadmin/agent evaluation set (not published; separate from the calibration data, ≈30 % of it with tool calls); pub. uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3), so anyone can reproduce it. Noise floor on the public trace (BF16 against itself): 0.0005. The W4A16 checkpoint was evaluated as a BF16 copy holding the dequantized INT4 × scale weights (Transformers, full load). Lower is better. Perplexity is on the same texts.
Context: largest --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak (262,144 fits without MTP).
Needle: 8 facts spread through one prompt of that length, all 8 returned.
Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases (fresh request, after a long story, full multi-turn session) × 20 tries, xhigh, at T=1.0 (no temperature sent, so vLLM uses the model's generation_config.json) and at T=0.3 (what opencode sends with the configuration below). The only misses (all at T=1.0) were answers taken from the history ("the folder is empty") instead of a new tool call. : 4 sysadmin tasks × 2 with and real tool execution (2026-10-02: the log-summary task failed in both runs; 7/8 on 2026-09-30).

Variants / revisions

Table with columns: Revision, MTP head, Size, Notes
Revision	MTP head	Size	Notes
`main`	BF16 (0.79 GiB)	14.6 GiB	reference; use without speculation for the full 262k context
`mtp-int8`	INT8 g128 (0.40 GiB)	14.2 GiB	recommended with MTP: same acceptance as BF16, +≈0.4 GiB for KV cache

Quantization recipe

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 128, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT4 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`,

Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens from HuggingFaceH4/ultrachat_200k (the first 512 of the train_sft split, shuffled with seed 42; chat template applied). Only the GPTQ linear layers use it: embed_tokens / lm_head (INT4) and the INT8 MTP head of the mtp-int8 revision are round-to-nearest, with no data.
Quantization took 26.7 min on a single RTX PRO 6000 Blackwell (96 GB).
Tokenizer and chat template are byte-identical to the base model. For tool calling we recommend the extra template chat_template-xml-tools.jinja (see Important notes).
Format: compressed-tensors (Marlin kernels in vLLM).

Requirements & install: vLLM with a small patch

Stock vLLM (checked on 0.29.0 and 0.30.0) does not pass quant_config to the embedding layer of qwen3_5 models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). The 4-line patch in patches/qwen3_5-embed-quant.patch fixes it for the main model and for the MTP module. The patch is taken from syv-ai/qwen38-27b-rtx3090 (Apache-2.0).

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.29.0" --torch-backend cu130 \    --extra-index-url https://flashinfer.ai/whl/ --index-strategy unsafe-best-matchVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

First download the tool-calling chat template (it is passed to vLLM as a local file):

bash

hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB chat_template-xml-tools.jinja --revision mtp-int8 --local-dir .

With MTP (recommended; mtp-int8 revision), text only, 222k context, 2 concurrent sequences (for a GPU used only for the LLM; 361 MiB really free at peak with the two memory settings, see Important notes):

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --language-model-only \  --max-model-len 222000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8640000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

With MTP and vision (mtp-int8 revision), 172k context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 176128 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7140000000 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

Without MTP, full 262k context, 4 concurrent sequences:

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB \  --language-model-only --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 9101639680 --max-num-seqs 4 --enable-prefix-caching \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

8 concurrent sessions with MTP (text, shared 200k KV pool): the 222k text command with --max-model-len 200000 --kv-cache-memory-bytes 8150000000 --max-num-seqs 8 (validated, 751 MiB really free at peak; see Benchmarks). With 8 sequences vLLM 0.29 runs out of memory while capturing CUDA graphs at ≥222k.

Important notes

Real VRAM headroom (corrected 2026-09-30): the driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use, and the "MiB free" figures in this card were computed as 24,576 − used, so they over-estimate the headroom by 453 MiB. Check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv. The original 222k command left only ≈50–70 MiB really free, and the same configuration crashed once with out-of-memory on the Swift 1.5 checkpoint (same recipe and size) during an agent request. The 222k command above now adds --max-num-batched-tokens 1024 and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: measured on this checkpoint the peak drops to 23,762 MiB (361 MiB really free) with the same prefill (586 tok/s at 205,792 tokens, 8/8 facts) and decode speed, no OOM. (The "really free" values in the 8-session rows of Benchmarks were read directly with nvidia-smi.)
For tool calling, use the base Qwen3.8 template with --tool-call-parser qwen3_coder (--chat-template chat_template-xml-tools.jinja). The original Qwopus V2 template asks for JSON tool calls (Hermes style), but the model often answers in Qwen3.8's XML format (<function=...><parameter=...>) or with malformed JSON, and the hermes parser silently drops those calls: the client receives an empty answer with finish_reason=stop. The two templates differ in how tools and tool calls are rendered (reasoning, vision and history are identical). An earlier version of this card recommended ; that was based on a short 3-tool test that did not reproduce real agent use.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). What was measured with opencode:

Tool calling with the real opencode request: the actual request that opencode sends (its long system prompt and its 11 tools), captured and replayed against vLLM, in three cases (fresh request, after a long story, full multi-turn session), several tries each.
End-to-end runs with opencode run: multi-step tasks (list files, count lines, read a file, write a summary) with real tool execution and streaming.

Context, speed and memory figures were measured directly against the vLLM API, not through opencode.

Start vLLM with --served-model-name qwopus3.8-27b (the model key in opencode must match it), then add a provider to ~/.config/opencode/config.json:

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwopus3.8-27b": {          "name": "Qwopus3.8-27B-Flash-V2 (local)",          "tool_call": true,          "temperature": true,          "limit": { "context": 222000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 0.3 },    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 0.3 }  }}

Parameters that matter:

limit.context = vLLM's --max-model-len (222,000 in the text configuration above). opencode compacts the history at about context − output, so output: 22000 makes it compact at ≈200k tokens of input and never exceed the server's limit.
Temperature 0.3 for the agents, as recommended by the Qwopus author for agentic/coding work (the repository default is 1.0). With the real opencode request: 15/15 tool calls at 0.3, 14/15 at 1.0 (20 tries per case on 2026-10-02: 60/60 at 0.3, 57/60 at 1.0).
Declare "temperature": true in the model. Without it opencode does not send any temperature to a custom model, even if the agent sets one (checked by capturing the requests).
Reasoning levels: the variants pass chat_template_kwargs; reasoning_effort accepts only xhigh, medium and low ( returns HTTP 400). They are prompt instructions and do not limit the amount of reasoning (see the notes above).

Benchmarks and validated configurations (single RTX 3090 24 GB)

Setup: vLLM 0.29.0 + patch, CUDA 13.0 wheels, driver 595 (CUDA 13.2), fp8 (e4m3) KV cache without calibrated scales, FlashInfer, --language-model-only, 370 W power limit (350 W for the 2026-10-02 runs). Decode is measured with 800 forced output tokens (ignore_eos).

Long context: 8 distinct facts spread from 3 % to 97 % depth, all 8 must be returned (greedy)

Table with columns: Configuration, Prompt tokens, Recovered, Time, Prefill
Configuration	Prompt tokens	Recovered	Time	Prefill
No MTP, 262k	57,874	8/8	59.8 s	968 tok/s
No MTP, 262k	246,276	8/8	435.2 s	566 tok/s
MTP 1 token (BF16 head), 208k	199,086	8/8	326.5 s	610 tok/s

Decode speed and MTP (Qwopus MTP head)

Table with columns: Configuration, Context, 1 request, t=0 / t=0.6, 4 concurrent, aggregate, Accepted tokens per step, Per-position acceptance
Configuration	Context	1 request, t=0 / t=0.6	4 concurrent, aggregate	Accepted tokens per step	Per-position acceptance
No speculation	262k	49.9 / 50.5 tok/s	177 tok/s	—	—
MTP, 1 token	208k	76.5 / 72.6 tok/s	287 tok/s	1.85	85 %
MTP, 2 tokens

Speculative decoding with MTP does not change the output: every drafted token is verified by the main model. Prefill cost of MTP: −1 to −3 %.

Prefix caching in an agentic session (≈90k-token history, each turn appends ≈2k tokens of tool output; time to first token)

Table with columns: Turn, 0, 1, 2, 3, 4, 5, 6
Turn	0	1	2	3	4	5	6
Without prefix caching	98.5 s	103.5 s	108.0 s	112.8 s	117.4 s	122.1 s	126.9 s
With `--enable-prefix-caching`	100.5 s	5.7 s

KV capacity cost of prefix caching: −2 % (270,663 → 265,356 tokens). With MTP (INT8) enabled as well, later turns take ≈6.8 s.

Concurrency (4 × 58k-token prompts at once, no MTP): 8/8 retrieval in all four, 988 tok/s aggregate prefill (same efficiency as one request).

VRAM (peak, 24,576 MiB total): 23,816 MiB without MTP (262k, 4 seqs); 23,592 MiB with MTP-INT8 (200k, 2 seqs, prefix caching).

Agent turns and 8 sessions (2026-10-02, 350 W, text, MTP 3; agent turn = the real opencode request, ≈13.5k → 14.5k tokens over 10 turns, with prefix caching; 8 sessions = --max-num-seqs 8, the KV pool shared by all):

Table with columns: Configuration, Turn: time to first token / total (median), First turn (no cache), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0 / 0.6), Really free (min)
Configuration	Turn: time to first token / total (median)	First turn (no cache)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0 / 0.6)	Really free (min)
150,000, KV 6.25e9	2.6 / 3.3 s	11.6 s	8 × 16.5k, 8/8 in all, 1,129 tok/s	462.8 / 447.3 tok/s	2,571 MiB
200,000, KV 8.15e9	2.6 / 3.3 s	11.6 s	8 × 22k, 8/8 in all, 1,105 tok/s	459.5 / 432.7 tok/s	751 MiB

Tool calling with a real coding-agent request (the actual request of the opencode agent: long system prompt, 11 tools; three cases — a fresh "list the files" request, the same after a long story, and a full multi-turn session — 5 tries each; 222k text configuration):

Table with columns: Sampling, Original template + hermes, Qwen3.8 template (chat_template-xml-tools.jinja) + qwen3_coder
Sampling	Original template + `hermes`	Qwen3.8 template (`chat_template-xml-tools.jinja`) + `qwen3_coder`
`reasoning_effort` xhigh, no temperature (model default 1.0)	3/15	14/15
xhigh, temperature 0.3	5/15	15/15
reasoning disabled, temperature 0.3	13/15	15/15
`reasoning_effort` low, temperature 0.3

With hermes, the failed answers contained XML tool calls (15), malformed JSON (10) or nothing (6; only in the session whose history already contained empty answers). The one failure with the XML template was not a format error: in the multi-turn session with empty answers in the history, the model claimed a result without calling the tool. With the XML template, multi-step tasks run from opencode (list, count lines, read a file, write a summary; 4–5 tool calls with streaming) succeeded 3/3. An earlier short test (10 sysadmin tasks, 3 tools) gave 10/10 with hermes; it did not reveal the problem.

Reasoning length (6 problems with checkable answers — arithmetic, word problem, counting, logic, subnetting, systemd — × 3 tries, temperature 0.6, top_p 0.95; reasoning tokens counted with /tokenize; total over 18 requests, compared with the base Qwen3.8 quantized with the same recipe):

Table with columns: reasoning_effort, Qwen3.8-27B (same recipe), Qwopus3.8-27B-Flash-V2, Correct (Qwen3.8 / Qwopus)
`reasoning_effort`	Qwen3.8-27B (same recipe)	Qwopus3.8-27B-Flash-V2	Correct (Qwen3.8 / Qwopus)
low	7,304	8,549	18/18 · 18/18
medium	8,592	9,534	18/18 · 18/18
xhigh	8,605	7,306	17/18 · 18/18

On these short problems both models reason about the same amount (±15 %, within the run-to-run spread), and the level does not regulate it. No answer hit the 6,000-token limit.

With 8 tries per problem (48 requests per level, temperature 0.6): 23,524 / 28,624 / 21,145 reasoning tokens for low / medium / xhigh on 2026-09-30 (47 · 46 · 48 correct, one answer cut at 6,000 tokens), and 21,854 / 23,177 / 20,852 on 2026-10-02 (48 · 48 · 48). Agent tasks with opencode (4 sysadmin tasks × 2, xhigh, temperature 0.3): 7/8 with 4,958 reasoning tokens (2026-09-30), 6/8 with 2,741 (2026-10-02). See the Swift 1.5 card for the three-model comparison.

Quality: KL divergence against BF16 (2026-10-02, exllamav3's qbench, method in Family comparison): internal evaluation set KLD 0.0357 (median 0.0129), perplexity 1.7436 vs 1.6706 for BF16; public trace 0.0251 (median 0.0024), perplexity 1.4707 vs 1.4288.

Vision (MTP 3 INT8 + prefix caching, --limit-mm-per-prompt '{"image":1,"video":0}', --mm-processor-kwargs '{"max_pixels":1003520}', 6.5 GiB KV cache → 173,559 tokens, 160k context). Synthetic test images with known ground truth:

Table with columns: Test, Result
Test	Result
Terminal screenshot of a failed `systemctl status` (1280×800)	correct service, exact error line (`bind() ... :8443 failed (98: Address already in use)`)
Chart with numeric labels (1280×640)	3/3 values, correct derived percentage
Dense `nft list ruleset` with 38 rules (1120×896, ≈1 MP)	3/3: found the one `backup-agent` rule (port, source network) and the default policy
Prefix caching with an image (second turn)	2.1 s → 1.5 s, correct follow-up answer
≈1 MP image + 115k-token text prompt	correct, peak VRAM 23,596 MiB (980 MiB free)

Images with the new 4-image battery (2026-10-02, vision configuration: 176,128 context, KV 7.14e9 B, MTP 3, --limit-mm-per-prompt '{"image":1,"video":0}'): 4/4 tasks, 18/18 checks (failed-service screenshot 3/3, chart 3/3, dense firewall ruleset 3/3, port held by another process with a dead PID as a trap 6/6), follow-up turn with prefix cache OK, 957 MiB really free at the end of the run (not a long-prompt peak), no engine errors.

Quality sanity checks (small in-house battery, temperature 0 unless noted): arithmetic, a multi-step word problem (with reasoning), a Python function with asserts and multi-turn memory were all correct. A free-form historical summary (Spanish) contained several factual errors; a factual question in Catalan about a local institution was answered correctly. The same battery on the sibling Qwen3.8-27B checkpoint gave the opposite pattern (better history, wrong location), so no winner can be declared from these checks.

Validated serving configurations (run as systemd services, 24 GB RTX 3090)

Each configuration below was started as a real service and checked with a near-maximum-length prompt.

Table with columns: Configuration, Context (KV pool), Checks, Peak VRAM (free)
Configuration	Context (KV pool)	Checks	Peak VRAM (free)
Text, MTP 3 (INT8 head), prefix caching, 2 seqs, XML template + `qwen3_coder` (dedicated GPU)	222,000 (223,441)	8/8 facts at 205,792 tokens (347 s, 593 tok/s prefill); decode 97.9 / 92.1 tok/s (t=0 / 0.6), 174 / 171 tok/s with 2 requests; ≈2.9 accepted tokens per step; real opencode request 59/60	24,070 MiB (506 MiB)
Vision + MTP 3 (INT8 head), prefix caching, 2 seqs	176,128 (178,923)	3/3 test images; image + 127k-token prompt correct (validated with the original template; the XML template only changes tool formatting)	23,752 MiB (824 MiB)
Text, no MTP, 4 seqs (`main`)	262,144 (271,342)

(The 222k row is the configuration before the two memory settings; with them the peak is 23,762 MiB, 361 MiB really free, see Important notes.)

Note: with MTP enabled, KV capacity grows in block-sized steps: always check the GPU KV cache size line after changing the KV bytes. For a GPU that also drives a display, use 8,350,000,000 bytes and --max-model-len 212800 (≈0.8 GB free at peak).

Alternative: DFlash 2 draft (tested, not recommended on one 24 GB GPU)

Instead of the MTP head, this checkpoint also works with the DFlash 2 draft z-lab/Qwen3.8-27B-DFlash2, quantized for 24 GB GPUs in JRamirez-UAB/Qwen3.8-27B-DFlash2-W4A16 (INT4 in main, INT8 in w8a16) — the draft was trained for the base Qwen3.8 but still accepts well on this fine-tune. Measured on one RTX 3090 (vLLM 0.29.0, 7 speculative tokens, fp8 KV cache, 2 sequences):

Table with columns: Speculation, Context (KV pool), Accepted tokens per step (decode · reasoning · tools), Decode, 1 request (t=0 / 0.6), 2 requests
Speculation	Context (KV pool)	Accepted tokens per step (decode · reasoning · tools)	Decode, 1 request (t=0 / 0.6)	2 requests
MTP 3 (INT8 head), this card's text configuration	222,000 (223,441)	≈2.9	98 / 92 tok/s	174 / 171 tok/s
DFlash 2 W4A16 draft, 6.3 GB KV cache	69,842	3.70 · 4.51 · 4.58	104 / 115 tok/s	169 / 165 tok/s
DFlash 2 W8A16 draft, 5.5 GB KV cache	60,854	3.75 · 4.78 · 4.97

The draft accepts more tokens per step, but on one GPU it gives only +10–25 % with one request and nothing with two, and its KV cache (≈90 KB per token against ≈35 KB with MTP in vLLM 0.29) limits the context to ≈60–70k tokens. The draft is meant for a two-GPU setup, where there is room for a long context; we do not have two GPUs, so that setup has not been tested. Details and commands on the draft's card.

Limitations

Quality has not been evaluated with standard benchmarks: only KL divergence against BF16, long-context retrieval, tool calling, vision, reasoning problems, agent tasks and a small sanity battery (above) were run on this checkpoint. Calibration used general chat data (ultrachat).
The INT4 RTN embeddings / lm_head are the main source of the KL divergence (KLD 0.036; our EXL3 3.85 bpw version of this model measures 0.0060).
The fp8 KV cache uses the default scale 1.0 (no calibrated KV scales). Long-context retrieval was nevertheless 8/8 up to 246k tokens.
All numbers come from one machine and one software stack; expect different figures on other drivers, power limits or vLLM versions.
Vision was validated with synthetic test images only (terminal screenshot, chart, dense text); not with natural photos or video.

Files and reproducibility

scripts/quantize.py: exact quantization script (llm-compressor); recipe.yaml: the recipe as serialized by llm-compressor.
scripts/mtp_quant.py: script that builds the mtp-int8 revision from main.
config.json.orig-llmcompressor: config as emitted by llm-compressor, before the two vLLM fixes described above.
chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
patches/: vLLM patch and its Apache-2.0 license.
SHA256SUMS: checksums of the model files.

Feedback welcome

I would be grateful for reports of installations of this checkpoint, both working and failed ones. Please open a discussion in the Community tab with your GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error message). Failed attempts are just as useful as successful ones: they help to document which setups work.

Credits & acknowledgements

This work is directly inspired by Marcin116 and their checkpoint Qwen3.8-27B-W4A16-AutoRound-fast-embed-int4, which showed that Qwen3.8-27B can serve very long contexts on RTX 3090-class GPUs by quantizing the token embeddings to INT4. We served and measured that checkpoint on our RTX 3090, and then set out to apply the same idea to the Qwopus V2 fine-tune.

What we took from that line of work:

the INT4 embed_tokens / lm_head layout (group 128, compressed-tensors pack-quantized), which is what makes the 262k context fit;
keeping the GatedDeltaNet in_proj_a / in_proj_b projections in BF16;
the vLLM patch that routes quantized embeddings through qwen3_5 and its MTP module.

Marcin116's checkpoint itself builds on born2bewild/Qwen3.8-27B-W4A16-AutoRound-fast and on the syvai/qwen3.8-27b-3090-fast-variant companion package (patch: syv-ai/qwen38-27b-rtx3090), developed with the Club3090 community. Many thanks to all of them. Any mistakes in this checkpoint are ours.

What is different here: a different base model (Qwopus3.8-27B-Flash-V2), GPTQ (llm-compressor) instead of AutoRound for the linear layers, the MTP head kept in BF16 or INT8 instead of INT4, and a full set of measurements on a single RTX 3090 (above).

Base model: Jackrong/Qwopus3.8-27B-Flash-V2, built on Qwen/Qwen3.8-27B.
INT4-embedding layout: Marcin116 / born2bewild; vLLM patch: syv-ai.
Tools: llm-compressor, compressed-tensors, vLLM.
KL-divergence evaluation: qbench from exllamav3 (turboderp) and its public evaluation trace.

License

Apache-2.0, same as the base model. The included patch is Apache-2.0 (syv-ai). This is a community quantization, not an official release of Jackrong or Qwen.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

Jackrong/Qwopus3.8-27B-Flash-V2

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Family comparison (Qwopus3.8-27B-Flash-V2, single RTX 3090 24 GB)

Table with columns: Model, Size, Context in 24 GB, KLD mean / median, Perplexity (BF16: 1.6706 · pub. 1.4288), Needle · tools at T=1.0 · T=0.3 · agent, Reasoning xhigh (tokens), Prefill, Decode with MTP, 1 / 2 sessions, Tokens per step, Accepted drafts (of 3)
Model	Size	Context in 24 GB	KLD mean / median	Perplexity (BF16: 1.6706 · pub. 1.4288)	Needle · tools at T=1.0 · T=0.3 · agent	Reasoning xhigh (tokens)	Prefill	Decode with MTP, 1 / 2 sessions	Tokens per step	Accepted drafts (of 3)
W4A16 GPTQ + INT4 embeddings (`mtp-int8`, this repo)	14.2 GiB	222,000 (text) · 176,128 (vision)	0.0357 / 0.0129 · pub. 0.0251 / 0.0024	1.7436 · pub. 1.4707	8/8 at 205,792 · 57/60 · 60/60 · 6/8	20,852	586 tok/s	97.9 / 174 tok/s	3.03–3.07	2.03–2.07
EXL3 3.85 bpw (`main`)	13.64 GiB

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 Qwopus3.8-27B-Flash-V2, measured with exllamav3's qbench. The first figure uses an internal sysadmin/agent evaluation set (not published; separate from the calibration data, ≈30 % of it with tool calls); pub. uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3), so anyone can reproduce it. Noise floor on the public trace (BF16 against itself): 0.0005. The W4A16 checkpoint was evaluated as a BF16 copy holding the dequantized INT4 × scale weights (Transformers, full load). Lower is better. Perplexity is on the same texts.
Context: largest --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak (262,144 fits without MTP).
Needle: 8 facts spread through one prompt of that length, all 8 returned.
Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases (fresh request, after a long story, full multi-turn session) × 20 tries, xhigh, at T=1.0 (no temperature sent, so vLLM uses the model's generation_config.json) and at T=0.3 (what opencode sends with the configuration below). The only misses (all at T=1.0) were answers taken from the history ("the folder is empty") instead of a new tool call. : 4 sysadmin tasks × 2 with and real tool execution (2026-10-02: the log-summary task failed in both runs; 7/8 on 2026-09-30).

Variants / revisions

Table with columns: Revision, MTP head, Size, Notes
Revision	MTP head	Size	Notes
`main`	BF16 (0.79 GiB)	14.6 GiB	reference; use without speculation for the full 262k context
`mtp-int8`	INT8 g128 (0.40 GiB)	14.2 GiB	recommended with MTP: same acceptance as BF16, +≈0.4 GiB for KV cache

Quantization recipe

Table with columns: Component, Treatment
Component	Treatment
Linear layers of the language model (attention, MLP, GatedDeltaNet `in_proj_qkv` / `in_proj_z` / `out_proj`)	GPTQ W4A16, symmetric INT4, group size 128, `actorder=static`, dampening 0.01
`embed_tokens`, `lm_head`	INT4 symmetric, group size 128, RTN (min-max)
GatedDeltaNet `in_proj_a`, `in_proj_b`, norms, `A_log`, `dt_bias`,

Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
Calibration: 512 samples × 2048 tokens from HuggingFaceH4/ultrachat_200k (the first 512 of the train_sft split, shuffled with seed 42; chat template applied). Only the GPTQ linear layers use it: embed_tokens / lm_head (INT4) and the INT8 MTP head of the mtp-int8 revision are round-to-nearest, with no data.
Quantization took 26.7 min on a single RTX PRO 6000 Blackwell (96 GB).
Tokenizer and chat template are byte-identical to the base model. For tool calling we recommend the extra template chat_template-xml-tools.jinja (see Important notes).
Format: compressed-tensors (Marlin kernels in vLLM).

Requirements & install: vLLM with a small patch

bash

uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.29.0" --torch-backend cu130 \    --extra-index-url https://flashinfer.ai/whl/ --index-strategy unsafe-best-matchVLLM_DIR=$(venv/bin/python -c 'import vllm,os;print(os.path.dirname(vllm.__file__))')patch -p1 -d "$VLLM_DIR" < patches/qwen3_5-embed-quant.patch

Quick start (single RTX 3090, 24 GB)

First download the tool-calling chat template (it is passed to vLLM as a local file):

bash

hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB chat_template-xml-tools.jinja --revision mtp-int8 --local-dir .

bash

PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --language-model-only \  --max-model-len 222000 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 8640000000 \  --max-num-batched-tokens 1024 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

With MTP and vision (mtp-int8 revision), 172k context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB --revision mtp-int8 \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 176128 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 7140000000 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

Without MTP, full 262k context, 4 concurrent sequences:

bash

vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-embed-int4-24GB \  --language-model-only --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 9101639680 --max-num-seqs 4 --enable-prefix-caching \  --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder \  --chat-template chat_template-xml-tools.jinja

Important notes

Real VRAM headroom (corrected 2026-09-30): the driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use, and the "MiB free" figures in this card were computed as 24,576 − used, so they over-estimate the headroom by 453 MiB. Check the real value with nvidia-smi --query-gpu=memory.used,memory.free --format=csv. The original 222k command left only ≈50–70 MiB really free, and the same configuration crashed once with out-of-memory on the Swift 1.5 checkpoint (same recipe and size) during an agent request. The 222k command above now adds --max-num-batched-tokens 1024 and PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True: measured on this checkpoint the peak drops to 23,762 MiB (361 MiB really free) with the same prefill (586 tok/s at 205,792 tokens, 8/8 facts) and decode speed, no OOM. (The "really free" values in the 8-session rows of Benchmarks were read directly with nvidia-smi.)
For tool calling, use the base Qwen3.8 template with --tool-call-parser qwen3_coder (--chat-template chat_template-xml-tools.jinja). The original Qwopus V2 template asks for JSON tool calls (Hermes style), but the model often answers in Qwen3.8's XML format (<function=...><parameter=...>) or with malformed JSON, and the hermes parser silently drops those calls: the client receives an empty answer with finish_reason=stop. The two templates differ in how tools and tool calls are rendered (reasoning, vision and history are identical). An earlier version of this card recommended ; that was based on a short 3-tool test that did not reproduce real agent use.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). What was measured with opencode:

Tool calling with the real opencode request: the actual request that opencode sends (its long system prompt and its 11 tools), captured and replayed against vLLM, in three cases (fresh request, after a long story, full multi-turn session), several tries each.
End-to-end runs with opencode run: multi-step tasks (list files, count lines, read a file, write a summary) with real tool execution and streaming.

Context, speed and memory figures were measured directly against the vLLM API, not through opencode.

Start vLLM with --served-model-name qwopus3.8-27b (the model key in opencode must match it), then add a provider to ~/.config/opencode/config.json:

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "qwopus3.8-27b": {          "name": "Qwopus3.8-27B-Flash-V2 (local)",          "tool_call": true,          "temperature": true,          "limit": { "context": 222000, "output": 22000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 0.3 },    "plan":  { "model": "local-vllm/qwopus3.8-27b", "variant": "xhigh", "temperature": 0.3 }  }}

Parameters that matter:

limit.context = vLLM's --max-model-len (222,000 in the text configuration above). opencode compacts the history at about context − output, so output: 22000 makes it compact at ≈200k tokens of input and never exceed the server's limit.
Temperature 0.3 for the agents, as recommended by the Qwopus author for agentic/coding work (the repository default is 1.0). With the real opencode request: 15/15 tool calls at 0.3, 14/15 at 1.0 (20 tries per case on 2026-10-02: 60/60 at 0.3, 57/60 at 1.0).
Declare "temperature": true in the model. Without it opencode does not send any temperature to a custom model, even if the agent sets one (checked by capturing the requests).
Reasoning levels: the variants pass chat_template_kwargs; reasoning_effort accepts only xhigh, medium and low ( returns HTTP 400). They are prompt instructions and do not limit the amount of reasoning (see the notes above).

Benchmarks and validated configurations (single RTX 3090 24 GB)

Long context: 8 distinct facts spread from 3 % to 97 % depth, all 8 must be returned (greedy)

Table with columns: Configuration, Prompt tokens, Recovered, Time, Prefill
Configuration	Prompt tokens	Recovered	Time	Prefill
No MTP, 262k	57,874	8/8	59.8 s	968 tok/s
No MTP, 262k	246,276	8/8	435.2 s	566 tok/s
MTP 1 token (BF16 head), 208k	199,086	8/8	326.5 s	610 tok/s

Decode speed and MTP (Qwopus MTP head)

Table with columns: Configuration, Context, 1 request, t=0 / t=0.6, 4 concurrent, aggregate, Accepted tokens per step, Per-position acceptance
Configuration	Context	1 request, t=0 / t=0.6	4 concurrent, aggregate	Accepted tokens per step	Per-position acceptance
No speculation	262k	49.9 / 50.5 tok/s	177 tok/s	—	—
MTP, 1 token	208k	76.5 / 72.6 tok/s	287 tok/s	1.85	85 %
MTP, 2 tokens

Speculative decoding with MTP does not change the output: every drafted token is verified by the main model. Prefill cost of MTP: −1 to −3 %.

Prefix caching in an agentic session (≈90k-token history, each turn appends ≈2k tokens of tool output; time to first token)

Table with columns: Turn, 0, 1, 2, 3, 4, 5, 6
Turn	0	1	2	3	4	5	6
Without prefix caching	98.5 s	103.5 s	108.0 s	112.8 s	117.4 s	122.1 s	126.9 s
With `--enable-prefix-caching`	100.5 s	5.7 s

KV capacity cost of prefix caching: −2 % (270,663 → 265,356 tokens). With MTP (INT8) enabled as well, later turns take ≈6.8 s.

Concurrency (4 × 58k-token prompts at once, no MTP): 8/8 retrieval in all four, 988 tok/s aggregate prefill (same efficiency as one request).

VRAM (peak, 24,576 MiB total): 23,816 MiB without MTP (262k, 4 seqs); 23,592 MiB with MTP-INT8 (200k, 2 seqs, prefix caching).

Table with columns: Configuration, Turn: time to first token / total (median), First turn (no cache), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0 / 0.6), Really free (min)
Configuration	Turn: time to first token / total (median)	First turn (no cache)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0 / 0.6)	Really free (min)
150,000, KV 6.25e9	2.6 / 3.3 s	11.6 s	8 × 16.5k, 8/8 in all, 1,129 tok/s	462.8 / 447.3 tok/s	2,571 MiB
200,000, KV 8.15e9	2.6 / 3.3 s	11.6 s	8 × 22k, 8/8 in all, 1,105 tok/s	459.5 / 432.7 tok/s	751 MiB

Table with columns: Sampling, Original template + hermes, Qwen3.8 template (chat_template-xml-tools.jinja) + qwen3_coder
Sampling	Original template + `hermes`	Qwen3.8 template (`chat_template-xml-tools.jinja`) + `qwen3_coder`
`reasoning_effort` xhigh, no temperature (model default 1.0)	3/15	14/15
xhigh, temperature 0.3	5/15	15/15
reasoning disabled, temperature 0.3	13/15	15/15
`reasoning_effort` low, temperature 0.3

Table with columns: reasoning_effort, Qwen3.8-27B (same recipe), Qwopus3.8-27B-Flash-V2, Correct (Qwen3.8 / Qwopus)
`reasoning_effort`	Qwen3.8-27B (same recipe)	Qwopus3.8-27B-Flash-V2	Correct (Qwen3.8 / Qwopus)
low	7,304	8,549	18/18 · 18/18
medium	8,592	9,534	18/18 · 18/18
xhigh	8,605	7,306	17/18 · 18/18

On these short problems both models reason about the same amount (±15 %, within the run-to-run spread), and the level does not regulate it. No answer hit the 6,000-token limit.

Table with columns: Test, Result
Test	Result
Terminal screenshot of a failed `systemctl status` (1280×800)	correct service, exact error line (`bind() ... :8443 failed (98: Address already in use)`)
Chart with numeric labels (1280×640)	3/3 values, correct derived percentage
Dense `nft list ruleset` with 38 rules (1120×896, ≈1 MP)	3/3: found the one `backup-agent` rule (port, source network) and the default policy
Prefix caching with an image (second turn)	2.1 s → 1.5 s, correct follow-up answer
≈1 MP image + 115k-token text prompt	correct, peak VRAM 23,596 MiB (980 MiB free)

Validated serving configurations (run as systemd services, 24 GB RTX 3090)

Each configuration below was started as a real service and checked with a near-maximum-length prompt.

Table with columns: Configuration, Context (KV pool), Checks, Peak VRAM (free)
Configuration	Context (KV pool)	Checks	Peak VRAM (free)
Text, MTP 3 (INT8 head), prefix caching, 2 seqs, XML template + `qwen3_coder` (dedicated GPU)	222,000 (223,441)	8/8 facts at 205,792 tokens (347 s, 593 tok/s prefill); decode 97.9 / 92.1 tok/s (t=0 / 0.6), 174 / 171 tok/s with 2 requests; ≈2.9 accepted tokens per step; real opencode request 59/60	24,070 MiB (506 MiB)
Vision + MTP 3 (INT8 head), prefix caching, 2 seqs	176,128 (178,923)	3/3 test images; image + 127k-token prompt correct (validated with the original template; the XML template only changes tool formatting)	23,752 MiB (824 MiB)
Text, no MTP, 4 seqs (`main`)	262,144 (271,342)

(The 222k row is the configuration before the two memory settings; with them the peak is 23,762 MiB, 361 MiB really free, see Important notes.)

Alternative: DFlash 2 draft (tested, not recommended on one 24 GB GPU)

Table with columns: Speculation, Context (KV pool), Accepted tokens per step (decode · reasoning · tools), Decode, 1 request (t=0 / 0.6), 2 requests
Speculation	Context (KV pool)	Accepted tokens per step (decode · reasoning · tools)	Decode, 1 request (t=0 / 0.6)	2 requests
MTP 3 (INT8 head), this card's text configuration	222,000 (223,441)	≈2.9	98 / 92 tok/s	174 / 171 tok/s
DFlash 2 W4A16 draft, 6.3 GB KV cache	69,842	3.70 · 4.51 · 4.58	104 / 115 tok/s	169 / 165 tok/s
DFlash 2 W8A16 draft, 5.5 GB KV cache	60,854	3.75 · 4.78 · 4.97

Limitations

Quality has not been evaluated with standard benchmarks: only KL divergence against BF16, long-context retrieval, tool calling, vision, reasoning problems, agent tasks and a small sanity battery (above) were run on this checkpoint. Calibration used general chat data (ultrachat).
The INT4 RTN embeddings / lm_head are the main source of the KL divergence (KLD 0.036; our EXL3 3.85 bpw version of this model measures 0.0060).
The fp8 KV cache uses the default scale 1.0 (no calibrated KV scales). Long-context retrieval was nevertheless 8/8 up to 246k tokens.
All numbers come from one machine and one software stack; expect different figures on other drivers, power limits or vLLM versions.
Vision was validated with synthetic test images only (terminal screenshot, chart, dense text); not with natural photos or video.

Files and reproducibility

scripts/quantize.py: exact quantization script (llm-compressor); recipe.yaml: the recipe as serialized by llm-compressor.
scripts/mtp_quant.py: script that builds the mtp-int8 revision from main.
config.json.orig-llmcompressor: config as emitted by llm-compressor, before the two vLLM fixes described above.
chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
patches/: vLLM patch and its Apache-2.0 license.
SHA256SUMS: checksums of the model files.

Feedback welcome

Credits & acknowledgements

What we took from that line of work:

the INT4 embed_tokens / lm_head layout (group 128, compressed-tensors pack-quantized), which is what makes the 262k context fit;
keeping the GatedDeltaNet in_proj_a / in_proj_b projections in BF16;
the vLLM patch that routes quantized embeddings through qwen3_5 and its MTP module.

Base model: Jackrong/Qwopus3.8-27B-Flash-V2, built on Qwen/Qwen3.8-27B.
INT4-embedding layout: Marcin116 / born2bewild; vLLM patch: syv-ai.
Tools: llm-compressor, compressed-tensors, vLLM.
KL-divergence evaluation: qbench from exllamav3 (turboderp) and its public evaluation trace.

License

Apache-2.0, same as the base model. The included patch is Apache-2.0 (syv-ai). This is a community quantization, not an official release of Jackrong or Qwen.