What is in this repository
Which one to use
All Qwopus3.8-27B-Flash-V2 variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):
Table with columns: Variant, Weights, Context on one RTX 3090, KLD vs BF16, Agent tasks, Decode, 1 request, Pick it for| Variant | Weights | Context on one RTX 3090 | KLD vs BF16 | Agent tasks | Decode, 1 request | Pick it for |
|---|
| GPTQ INT4 | 14.21 GiB | 222,000 (text) · 176,128 with vision | 0.0357 | 29/30 | 114.3 tok/s | the longest context on stock vLLM (+ a 4-line patch): 222,000 text-only, or 176,128 with vision |
| GPTQ INT8 g64 (this repo) | 15.75 GiB | 176,000 (text) · 131,072 with vision | 0.0145 | 25/30 | 97.3 tok/s | a more faithful GPTQ (KLD 0.0145) on stock vLLM (+ the same patch), with less context |
| EXL3 3.85 | 13.62 GiB | 215,000 with vision | 0.0060 | 27/30 | 86.9 tok/s | the closest to BF16 (KLD 0.0060) with vision at 215,000; needs exllamav3 + our plugin |
| EXL3 3.70 | 13.19 GiB | 226,000 with vision | 0.0073 | 25/30 | 83.9 tok/s | more context (226,000) and 4 sessions with vision, a little less faithful than 3.85 |
Files
model.safetensors (+ index), model_mtp.safetensors (INT8 MTP head), config.json (compressed-tensors), recipe.yaml (as serialized by llm-compressor).
scripts/quantize_v4.py, scripts/mtp_quant.py; patches/ (vLLM patch and its Apache-2.0 license); SHA256SUMS.
chat_template-xml-tools.jinja: the base Qwen3.8 chat template (XML tool calls), recommended for tool calling (Apache-2.0, Qwen).
Results
Family comparison (Qwopus3.8-27B-Flash-V2, same conditions)
Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (GPTQ: + the embedding patch / EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).
Table with columns: Variant, Weights, KLD mean / median (internal · public), Perplexity (BF16 1.6706 · pub. 1.4288), Needle (8 facts) near max. context, Tools: real opencode request, Agent tasks (10 × 3), Reasoning xhigh: correct · tokens, Vision| Variant | Weights | KLD mean / median (internal · public) | Perplexity (BF16 1.6706 · pub. 1.4288) | Needle (8 facts) near max. context | Tools: real opencode request | Agent tasks (10 × 3) | Reasoning xhigh: correct · tokens | Vision |
|---|
GPTQ W4A16 + INT4 embeddings (mtp-int8) | 14.21 GiB | 0.0357 / 0.0129 · pub. 0.0251 / 0.0024 | 1.7436 · pub. 1.4707 | 8/8 at 204,551 | 55/60 | 29/30 | 47/48 · 18,090 | 15/15 |
GPTQ W4A16 g64 + INT8 embeddings (main, this repo) | 15.75 GiB | 0.0145 / 0.0047 · pub. 0.0115 / 0.0011 | 1.7029 · pub. 1.4507 | 8/8 at 162,208 | 57/60 | 25/30 | 48/48 · 17,965 | 15/15 |
EXL3 3.85 bpw (main) | 13.62 GiB | 0.0060 / 0.0018 · pub. 0.0048 / 0.0004 | 1.6815 · pub. 1.4379 | 8/8 at 197,991 | 54/60 | 27/30 | 48/48 · 23,460 | 15/15 |
EXL3 3.70 bpw (3.70bpw) | 13.19 GiB | 0.0073 / 0.0022 · pub. 0.0060 / 0.0005 | 1.6856 · pub. 1.4409 | 8/8 at 208,158 | 57/60 | 25/30 | 48/48 · 26,996 | 15/15 |
Table with columns: Variant, Context on one RTX 3090, Sessions, Prefill near max. context, Decode, 1 / 2 / 8 requests (tok/s), MTP tokens per step, VRAM really free at peak| Variant | Context on one RTX 3090 | Sessions | Prefill near max. context | Decode, 1 / 2 / 8 requests (tok/s) | MTP tokens per step | VRAM really free at peak |
|---|
| GPTQ INT4 | 222,000 (text) · 176,128 with vision | 2 | 603 tok/s | 114.3 / 192.2 / 189.5 | 3.20 | 284 MiB · vision config: 930 MiB |
| GPTQ INT8 g64 | 176,000 (text) · 131,072 with vision | 2 | 644 tok/s | 97.3 / 173.0 / 175.1 | 3.21 | 196 MiB · vision config: 840 MiB |
| EXL3 3.85 | 215,000 with vision | 2 | 554 tok/s | 86.9 / 140.5 / 145.3 | 3.40 | 320 MiB |
| EXL3 3.70 | 226,000 with vision | 4 | 543 tok/s | 83.9 / 145.7 / 258.8 | 3.21 | 148 MiB |
How to read it:
- KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's
qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign.
- Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (
generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test.
- Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run.
- VRAM really free: lowest
memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB.
- Older sections below keep their own date and setup; use this table to compare variants.
How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):
Table with columns: Variant, KV pool (tokens), Max sessions, 32k each, 64k each, 128k each, Full context each| Variant | KV pool (tokens) | Max sessions | 32k each | 64k each | 128k each | Full context each |
|---|
| GPTQ INT4 | 223,441 | 2 | 2 | 2 | 1 | 1 × 222,000 |
| GPTQ INT4 (vision config) | 178,923 | 2 | 2 | 2 | 1 | 1 × 176,128 |
| GPTQ INT8 g64 | 178,816 | 2 | 2 | 2 | 1 | 1 × 176,000 |
| GPTQ INT8 g64 (vision config) | 132,423 | 2 | 2 | 2 | 1 | 1 × 131,072 |
| EXL3 3.85 | 216,433 | 2 | 2 | 2 | 1 | 1 × 215,000 |
| EXL3 3.70 | 231,757 | 4 | 4 | 3 | 1 | 1 × 226,000 |
Few users or many users. The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.
Table with columns: Variant, Configuration, Decode by simultaneous requests (aggregate tok/s), N sessions at once, needle 8/8, Tools (5 × 3), VRAM really free| Variant | Configuration | Decode by simultaneous requests (aggregate tok/s) | N sessions at once, needle 8/8 | Tools (5 × 3) | VRAM really free |
|---|
| GPTQ INT4 | Quick start (2 sessions, MTP 3) | 1: 114 · 2: 192 · 8: 190 | — | 55/60 | 284 MiB |
| GPTQ INT4 | many users (8 sessions of up to 222,000 tokens, MTP 3) | 1: 113 · 2: 190 · 4: 340 · 8: 481 | 8/8 of 22k | 15/15 | 174 MiB |
| GPTQ INT8 g64 | Quick start (2 sessions, MTP 3) | 1: 97 · 2: 173 · 8: 175 | — | 57/60 | 196 MiB |
| GPTQ INT8 g64 | many users (8 sessions of up to 165,000 tokens, MTP 3) | 1: 96 · 2: 166 · 4: 316 · 8: 460 | 8/8 of 22k | 14/15 | 386 MiB |
| EXL3 3.85 | Quick start (2 sessions, MTP 3) | 1: 87 · 2: 140 · 8: 145 | — | 54/60 | 320 MiB |
| EXL3 3.85 | many users (8 sessions of up to 215,000 tokens, MTP 3) | 1: 88 · 2: 142 · 4: 257 · 8: 314 | 8/8 of 22k | 14/15 | 138 MiB |
| EXL3 3.70 | Quick start (4 sessions, MTP 3) | 1: 84 · 2: 146 · 8: 259 | — | 57/60 | 148 MiB |
| EXL3 3.70 | many users (8 sessions of up to 220,000 tokens, MTP 3) | 1: 78 · 2: 143 · 4: 254 · 8: 293 | 8/8 of 22k | 15/15 | 358 MiB |
Earlier measurements
Earlier measurements (single RTX 3090 24 GB, 300 W, 2026-10-06), before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.
Table with columns: Test, Result| Test | Result |
|---|
| KV pool | 178,816 tokens (--max-model-len 176000, KV 7.09e9 B) |
| 8 facts spread over 161,646 tokens (greedy) | 8/8, 252 s (643 tok/s prefill) |
| 2 concurrent sessions of 79k tokens | 8/8 and 8/8, 839 tok/s aggregate prefill |
| Decode, 1 request, t=0 / t=0.6 | 85.4 / 83.8 tok/s |
| Decode, 2 concurrent, aggregate, t=0 / t=0.6 | 166.6 / 151.3 tok/s |
| MTP mean acceptance length (tokens per step, max 4) | 3.57 |
Real opencode request, tool calls (xhigh, 3 cases × 5) | 15/15, no parse errors |
| Reasoning tokens, 6 checkable problems × 8 (T 0.6), correct | low 18,047 (48/48) · medium 25,190 (47/48) · xhigh 17,698 (48/48) |
opencode run, 4 sysadmin tasks × 2 (xhigh, T 0.3) | 7/8 tasks, 4,313 reasoning tokens, 290 s, 45 requests |
| Peak VRAM / really free | 23,854 MiB / 271 MiB, no OOM; fresh install from this repo: 177 MiB free |
| Installed from this repo following this card (2026-10-09) | 8/8 at 154,854 tokens, 2 × 76k 8/8, tool calls 14–15/15, no engine errors |
Vision (the quick-start command without --language-model-only, with --max-model-len 131072 --kv-cache-memory-bytes 5500000000 --limit-mm-per-prompt '{"image":1,"video":0}' --mm-processor-kwargs '{"max_pixels":1003520}' and the XML template): KV pool 132,423 tokens; 4-image battery 15/15, follow-up turn with prefix cache OK; needle 8/8 at 115,483 tokens; tool calls with the real opencode request 15/15; 115 MiB really free at peak.
Quality: KL divergence against BF16 (2026-10-05)
Measured with exllamav3's qbench, first on an internal sysadmin/agent evaluation set (not published; different conversations from the
calibration trace, but from the same domain), the W4A16 checkpoint evaluated as a BF16 copy holding the dequantized weights (Transformers):
Table with columns: Checkpoint, KLD mean, median, p90, Perplexity (BF16 1.6706)| Checkpoint | KLD mean | median | p90 | Perplexity (BF16 1.6706) |
|---|
| This repo (v4) | 0.0145 | 0.0047 | 0.0395 | 1.7029 |
| INT4 embeddings, g128, UltraChat calibration | 0.0357 | 0.0129 | — | 1.7436 |
On turboderp's public evaluation trace (qbench_prompts_gen.json, the one used in all our cards, unrelated to our calibration data) the gain
holds: KLD 0.0115 (median 0.0011, perplexity 1.4507) against 0.0251 (median 0.0024) for the INT4-embedding checkpoint,
so it is not just an effect of calibrating and evaluating on the same domain.
On Swift 1.5 we isolated the steps (internal set): INT8 embeddings / lm_head 0.0467 → 0.0396, + mixed calibration 0.0251, + group size 64 0.0215.
Sampling and repetition loops
Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 3.85 build of Qwopus3.8-27B-Flash-V2 (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.
Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Temperature | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| Loops / 30 | 3 | 1 | 2 | 1 | 2 | 0 | 0 | 0 |
| Answers that finished before 16,000 tokens / 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 | 30 |
From temperature 0.8 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.
Other models, same test and setup (loops / 30):
Table with columns: Model (build), 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Model (build) | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| Qwen3.8-27B · EXL3 3.85 | 12 | 12 | 7 | 8 | 1 | 1 | 1 | 0 |
| Qwen3.8-27B-pi · EXL3 3.85 | 9 | 4 | 9 | 10 | 4 | 2 | 1 | 0 |
| Swift 1.5 (Qwen3.8-27B) · EXL3 3.85 | 7 | 5 | 3 | 4 | 3 | 1 | 1 | 0 |
| Ornith-1.5-9B · EXL3 6.00 (16 GB) | 6 | 2 | 3 | 1 | 1 | 0 | 0 | 0 |
| NeoHorse-1-9B · EXL3 6.00 (16 GB) | 6 | 3 | 3 | 1 | 0 | 0 | 0 | 0 |
| Qwen3.6-35B-A3B · EXL3 4.00 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
| Ornith-1.5-35B-A3B · EXL3 4.00 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Qwopus3.6-35B-A3B-Coder · EXL3 4.00 | 9 | 13 | 12 | 6 | 11 | 6 | 5 | 4 |
| Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM | 24 | 24 | 21 | 21 | 17 | 8 | 0 | 0 |
| Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM | — | — | 13 | 7 | 5 | | | |
Requirements & install (vLLM + vllm-quantpack)
Before you install (checked on a clean Ubuntu 24.04 host with an A40, 2026-10-28):
- An NVIDIA driver that supports CUDA 13.2 (tested with 595.84) and uv.
- The Python 3.12 headers and a C compiler.
uv venv --python 3.12 uses the system Python 3.12 when there is one, and Triton
compiles a small C helper every time vLLM starts; without the headers every vllm serve stops with
fatal error: Python.h: No such file or directory. On Debian/Ubuntu: sudo apt-get update && sudo apt-get install -y python3.12-dev build-essential.
- No CUDA toolkit is needed: the FlashInfer line below installs prebuilt kernels (without it, vLLM looks for
nvcc at start and fails with
Could not find nvcc and default cuda_home='/usr/local/cuda' doesn't exist).
- Disk: about 20 GB for the venv, as much again for uv's cache (
uv cache clean frees it), plus the model.
Stock vLLM (0.29 to 0.31) does not pass quant_config to the embedding layer of qwen3_5-family models, so a checkpoint with quantized embeddings fails to load (no parameter named 'embed_tokens.weight_packed'). Our vllm-quantpack plugin (on PyPI) fixes that from outside vLLM, without patching it:
uv venv --python 3.12 venv && uv pip install --python venv/bin/python "vllm==0.31.0" --torch-backend=auto # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \ --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match # FlashInfer kernelsuv pip install --python venv/bin/python vllm-quantpackvenv/bin/hf download JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-g64-embed-int8-24GB chat_template-xml-tools.jinja --local-dir . # Qwen3.8 XML tool-call template (see Important notes)
Plugin (2026-10-28): the 4-line vLLM patch is replaced by vllm-quantpack (pip install vllm-quantpack), which loads the quantized embeddings on unpatched vLLM. The measurements in this card were made with the previous stack (vLLM 0.31.0 + the embedding patch); this checkpoint was then served again with vllm-quantpack 0.1.0 following this card on a clean install (text, image and tool calls checked). The previous files stay in patches/ for reference.
Quick start (single RTX 3090, 24 GB)
Text only, MTP 3, 176k context, 2 concurrent sequences (validated configuration):
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \vllm serve JRamirez-UAB/Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-g64-embed-int8-24GB \ --served-model-name qwopus3.8-27b \ --language-model-only \ --max-model-len 176000 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 7090000000 \ --max-num-batched-tokens 1024 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder \ --chat-template chat_template-xml-tools.jinja
For many users (measured on 2026-10-14, same GPU): the same command with --kv-cache-memory-bytes 6790000000 --max-model-len 165000 --max-num-seqs 8 — 8 sessions of up to 165,000 tokens, MTP 3. Aggregate decode 460 tok/s with 8 requests (Quick start: 175 with 8); a single request runs at 96 tok/s (97 with the Quick start); 8 of 8 simultaneous sessions returned all 8 needle facts; 386 MiB of VRAM really free. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.
Using it with opencode
How we use it as a coding / sysadmin agent from opencode (opencode 1.18.x). limit.context must match --max-model-len of your
vllm serve command (176000 in the Quick start above).
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "qwopus3-8-27b-flash-v2-gptq-w4a16-g64-em": { "name": "Qwopus3.8-27B-Flash-V2-GPTQ-W4A16-g64-embed-int8-24GB (local)", "tool_call": true, "temperature": true, "limit": { "context": 176000, "output": 22000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/qwopus3-8-27b-flash-v2-gptq-w4a16-g64-em", "variant": "xhigh", "temperature": 1.0 }, "plan": { "model": "local-vllm/qwopus3-8-27b-flash-v2-gptq-w4a16-g64-em", "variant": "xhigh", "temperature": 1.0 } }}
- Declare
"temperature": true in the model, otherwise opencode sends no temperature to a custom model and vLLM falls back to its defaults.
- Temperature 1.0 is this repo's
generation_config.json default. Our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and in our agent tests (10 real sysadmin tasks × 3 runs with opencode run) 1.0 scored the same as 0.3 within one task: 29/30 at 1.0 and 30/30 at 0.3 for Qwen3.8-27B, -pi and Swift 1.5 27B; Qwopus3.8, served with its chat_template-xml-tools.jinja, 28/30 at 1.0 and 26/30 at 0.3.
- With
output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
- Start opencode inside a project directory (it takes a git snapshot before each step); for
opencode run from scripts, close stdin (</dev/null).
Important notes
Memory (read this before changing the numbers)
- The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use. "24,576 − used" over-estimates the headroom by that amount;
check the real value with
nvidia-smi --query-gpu=memory.used,memory.free --format=csv.
- The configuration above left 177 MiB really free at peak when installed exactly as described here (fresh venv, model downloaded from this repo,
155k-token prompt and 2 × 76k concurrent); our own build peaked at 23,854 MiB used / 271 MiB free (415 MiB free at rest).
Both are above our 100 MiB safety margin, but there is little room: do not raise the KV cache.
--max-num-batched-tokens 1024 and expandable_segments are part of that budget: without them the prefill peak is several hundred MiB higher.
- Size the KV cache with
--kv-cache-memory-bytes, not --gpu-memory-utilization: with this hybrid (GatedDeltaNet) model the memory profiling
under-estimates the prefill activations and a percentage over-allocates the KV cache.
- With MTP the KV capacity grows in block-sized steps: check the
GPU KV cache size line; --max-model-len must not exceed it.
- Our RTX 3090 hung once (driver
GSP RM heartbeat timed out, Xid 154) during a 168k-token prefill of this checkpoint at a 350 W power limit;
at 300 W with a more aggressive fan curve the same test passed and all the numbers here are from that setup. If your card is marginal, lower its power limit.
Qwopus' own chat template asks for JSON tool calls (hermes), but the model often answers in Qwen3.8's XML format and hermes then drops the call
silently. With the base Qwen3.8 template chat_template-xml-tools.jinja and --tool-call-parser qwen3_coder
the real opencode request gave 59/60 on the INT4-embedding sibling and 15/15 on this checkpoint. Always pass --chat-template chat_template-xml-tools.jinja.
Other notes
- Use
--tool-call-parser qwen3_coder. On Ampere (sm86), fp8 KV cache only works with the FlashInfer attention backend (the default).
reasoning_effort accepts only xhigh, medium and low (high returns HTTP 400); they are prompt instructions, not limits. For a hard
cap send thinking_token_budget in the request (vLLM 0.29 enforces it with --reasoning-parser qwen3, also with MTP).
- For use from opencode, the configuration in the INT4-embedding card applies with
"limit": { "context": 176000, "output": 22000 } (declare "temperature": true in the model, or opencode sends no temperature).
Quantization recipe ("v4")
Table with columns: Component, Treatment| Component | Treatment |
|---|
Linear layers of the language model (attention, MLP, GatedDeltaNet in_proj_qkv / in_proj_z / out_proj) | GPTQ W4A16, symmetric INT4, group size 64, actorder=static, dampening 0.01 |
embed_tokens, lm_head | INT8 symmetric, group size 128, RTN (min-max) |
GatedDeltaNet in_proj_a, in_proj_b, norms, A_log, dt_bias, conv1d | kept in BF16 |
MTP head (mtp.*) | INT8 g128 RTN (mtp.fc, mtp.layers.*); shares the INT8 embeddings / lm_head (the source model keeps Qwen3.8-27B's MTP head unchanged, so this file is the same as in our Qwen3.8 v4 repo) |
| Vision tower | kept in BF16 |
- Source:
Jackrong/Qwopus3.8-27B-Flash-V2 revision 13f92e09.
- Tool: llm-compressor 0.14.0, compressed-tensors 0.19.0, transformers 5.17.0, torch 2.14.0+cu130.
- Calibration: 512 samples × 2048 tokens: 196 conversations from an internal Linux sysadmin / agent trace (multi-turn, with tool calls and reasoning;
not published) plus 316 samples of
HuggingFaceH4/ultrachat_200k (train_sft, seed 42), shuffled. Only GPTQ uses it; the INT8 parts are RTN.
- Tokenizer and chat template are identical to the base model (for tool calling use the Qwen3.8 XML template shipped here, see Important notes).
- Scripts:
scripts/quantize_v4.py (with CAL_TRACE you can mix in your own trace; without it, UltraChat only) and
scripts/mtp_quant.py (INT8 MTP head).
Integrity and validation
What was checked for this checkpoint, how, and where the numbers are (measured on 2026-10-14; the method and every figure are in Family comparison).
Artifact integrity
- Every weight file is checked against our local copy with SHA-256 before publication (
SHA256SUMS in this repo).
- The measurements used the weights downloaded from this repo at commit
5449c2e65c67; later commits change only metadata, documentation or the plugin, never the .safetensors files.
- vLLM loads every tensor listed in
model.safetensors.index.json on start-up (a missing tensor or a wrong shape would stop it): checked from a fresh download.
Quantization
- Quantization recipe and calibration in
recipe.yaml and scripts/ (llm-compressor GPTQ, compressed-tensors format).
- Fidelity: KL divergence against the BF16 Qwopus3.8-27B-Flash-V2 with exllamav3's
qbench on two traces (ours and turboderp's public one).
Chat template
- The published template renders correctly in 40 edge cases (empty and very long system prompt, no tools, two tool calls in one message, nested JSON and multi-line arguments, images with tools, two images, earlier reasoning, multi-turn with several tool results) × the reasoning settings (
enable_thinking: false, reasoning_effort low / xhigh): all pass (prueba_plantillas.py). With Transformers directly, pass tool-call arguments as dicts (vLLM converts JSON strings itself).
Runtime
- Load, long-context needle, vision, tool calls with the real opencode request, 10 agent tasks × 3 runs, concurrency and VRAM: see Family comparison; engine errors during the run: 0.
- On a desktop RTX 3090 that also drives the display (300 W): Quick start at 176,000 tokens: 193 MiB really free; vision config at 131,072 tokens: 891 MiB really free.
Scope: results apply to the revision, software stack (vLLM 0.31.0 + the embedding patch), hardware and settings stated above; other setups are not tested.
Limitations
- Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with
sudo, and give the agent a non-privileged account and a test machine where you can.
- Quality was not evaluated with standard benchmarks; the checks are KL divergence (above), long-context retrieval, tool calling, short reasoning
problems and agent tasks.
- 176k is less than the 262k the model supports; with 24 GB, use the INT4-embedding or EXL3 checkpoints for longer contexts.
- The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.
Feedback welcome
Reports of installations, both working and failed ones, are very welcome in the
Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
Credits & license