What is in this repository
Which one to use
Table with columns: GPU, Build, Context per session, Real concurrent sessions of 236k, KV cache, Pick it for| GPU | Build | Context per session | Real concurrent sessions of 236k | KV cache | Pick it for |
|---|
| 24 GB (RTX 3090) | this repo (3.50 bpw) | 262,144 (text) | 1 (pool 377,712 tokens) | int4 | the best quality on 24 GB: agent, tools, fewest loops |
| 16 GB | — | — | — | — | does not fit: 14.99 GB of weights |
Our 2.25 bpw build of the same model holds 2 sessions of 236k on the same card, but in our tests its KL divergence is about 7 times higher, it passed 25/30 agent tasks and it loops on long answers even at temperature 1.0 (see Comparison): use it only when two full-context sessions on one card matter more than quality.
Files and reproducibility
Table with columns: File, What it is| File | What it is |
|---|
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json | EXL3 weights (14.99 GB) and their index |
quantization_config.json | exllamav3's per-tensor storage record (bits per tensor; calibration size only, no text) |
config.json, generation_config.json | model config; sampling defaults of the base model |
chat_template.jinja, tokenizer.json, tokenizer_config.json | chat template and tokenizer of the base model, unchanged |
preprocessor_config.json, processor_config.json, video_preprocessor_config.json | image/video processor of the base model |
SHA256SUMS | SHA-256 of every file above |
Not included: the BF16 reference, the calibration data (internal), the plugin (install it from PyPI, see below).
Results
All tables: one RTX 3090 24 GB (no other use), vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4, --max-model-len 262144, --kv-cache-dtype int4_per_token_head, text only, the Quick start configuration, measured on 2026-11-02 unless stated otherwise.
Quantization quality (exllamav3 qbench, against BF16)
Table with columns: Trace, KLD mean, KLD median, KLD p90, Perplexity (BF16)| Trace | KLD mean | KLD median | KLD p90 | Perplexity (BF16) |
|---|
| opencode agent trace (ours) | 0.0127 | 0.0039 | 0.035 | 2.6141 (2.5982) |
| sysadmin trace (ours) | 0.0148 | 0.0044 | 0.042 | 1.6571 (1.6332) |
public (qbench_prompts_gen.json, turboderp) | 0.0104 | 0.0011 | 0.032 | 1.4331 (1.4128) |
Capacity on one RTX 3090 (24 GB)
Table with columns: --kv-cache-memory-bytes, KV pool reported by vLLM, Real concurrent sessions of 235,929 tokens, Peak used, Really free at peak--kv-cache-memory-bytes | KV pool reported by vLLM | Real concurrent sessions of 235,929 tokens | Peak used | Really free at peak |
|---|
| 6914897346 | 377,712 tokens | 1 (with 2 requested, the second waits; 0 preemptions) | 22,160 MiB | 1,956 MiB |
The token embeddings are stored in BF16 (2.54 GB); that is what limits the pool. Shorter sessions share it: by vLLM's count, 2 sessions of up to 188k or 8 of up to 47k fit at once (not load-tested at those sizes).
Test battery
Table with columns: Test, Machine A, Machine B| Test | Machine A | Machine B |
|---|
| Tool calls, battery | 30/30 valid and correct, 0 errors | 30/30 |
Tool calls, real opencode request, 3 cases × 20 (xhigh) | 20/20, 19/20, 20/20 = 59/60, 0 invalid | 60/60 |
Agent tasks, opencode run, 10 sysadmin tasks × 3 runs, temperature 1.0 | 10/10, 10/10, 10/10 | 10/10, 10/10, 10/10 |
| Programming tasks (12 × 3) | 12/12, 12/12, 12/12 (1,886-2,348 s per run) | — |
| Needle, 8 facts in one prompt (another KV budget, 5.0e9 B) | — | 64k 8/8 · 128k 8/8 · 245k 8/8 · 258k 8/8 |
| Prefill / decode near 15k · 64k · 128k · 260k | 1,115 / 43.0 · 972 / 37.5 · 815 / 32.3 · 616 / 25.4 tok/s | — |
| Agents at once, tasks passed (runs that reached the 600 s limit) | 4: 6/8 (2) · 8: 13/16 (5) · 16: 26/32 (14) | — |
| Time to first token of a new 32k prompt (no cache / cached), with 4 · 8 · 16 agents running | 30.0 / 1.8 s · 30.3 / 1.9 s · 34.0 / 2.6 s | — |
Both machines are RTX 3090 24 GB with the same software stack. With sampling on, the same checkpoint varies by about ±2 of 30 between runs.
Comparison: the five Qwopus3.8-Coder-Flash builds we tested
Same model, two bit rates and three calibration choices; same card, settings and tests as above (agent and tool numbers: machine A).
Table with columns: Build, KLD opencode / public / sysadmin, Real 236k sessions, Programming, Loops /30 at T 0.6 / 0.8 / 1.0, Tools (30), Agent (3 × 10)| Build | KLD opencode / public / sysadmin | Real 236k sessions | Programming | Loops /30 at T 0.6 / 0.8 / 1.0 | Tools (30) | Agent (3 × 10) |
|---|
| 2.25, no calibration data | 0.0964 / 0.0860 / 0.1205 | 2 | 35/36 | 24 / 16 / 21 | 27 | 27/30 |
| 2.25, calibration as this repo (2.25 bpw build) | 0.0888 / 0.0798 / 0.1087 | 2 | 35/36 | 14 / 13 / 8 | 29 | 25/30 |
| 2.25, steps written by Qwopus | 0.0869 / 0.0810 / 0.1081 | 2 | 36/36 | 26 / 16 / 10 | 30 | 27/30 |
| 3.50, calibration as this repo (this repo) | 0.0127 / 0.0104 / 0.0148 | 1 | 36/36 | 12 / 6 / 0 | 30 | 30/30 |
| 3.50, steps written by Qwopus | 0.0126 / 0.0105 / 0.0152 | 1 | 35/36 | 18 / 7 / 2 | 30 | 28/30 |
At 3.50 bpw the calibration source changes the KL divergence by less than 3 %; at 2.25 the calibration data lowers it by 8-10 %.
Sampling and repetition loops
5 long Spanish sysadmin questions × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated word 8-grams.
Table with columns: Temperature, 0.6, 0.8, 1.0| Temperature | 0.6 | 0.8 | 1.0 |
|---|
| Loops / 30 | 12 | 6 | 0 |
| Answers that finished before 16,000 tokens / 30 | 11 | 10 | 11 |
Use temperature 1.0 (the default in generation_config.json). Below 0.8 this model loops often on long answers, as other Qwen3.8-27B fine-tunes do. Many long answers did not finish within 16,000 tokens even without a loop: this model writes long answers, so set a max_tokens that suits you.
Known model behaviour and discarded options
- Speed: a dense 27B; one request decodes at 43 tok/s near 15k tokens and 25 tok/s near 260k on a 3090, so long agent tasks take minutes. With 16 agents at once on one card many tasks reach a 10-minute limit.
- Temperature 0 loops: greedy decoding loops inside the reasoning on long answers, in BF16 as well as in this build (see the warning at the top).
- Vision and MTP (measured after the text-only tests): without
--language-model-only and with --speculative-config '{"method":"mtp","num_speculative_tokens":3}' at 262,144 context on one RTX 3090, the KV pool is 382,730 tokens (one full-context session, 22,000 MiB peak). Vision 15/15 and tool calls 30/30 in both server starts; MTP accepted 2.09 tokens per draft on short answers and 2.74 near 258k; single-request decode 82 tok/s on a short cached prompt; 1/2/4/8 requests: 76 / 128 / 206-228 / 223 tok/s. The 258k prefill took 468 s. With MTP on, vLLM 0.31 can only reuse a cached long prompt when the new prompt contains all of it (see Important notes); that, not vision, is what slowed our long-context run, so no long-context decode number is shown.
- Capacity alternative: the 2.25 bpw build holds 2 sessions instead of 1, with about 7 times the KL divergence and many more loops; the other two 2.25 builds in Comparison are not published.
Requirements & install (vLLM 0.31 + exllamav3 1.6.5 + vllm-quantpack 0.1.4)
Before you install (the same stack as our other cards, checked on a clean Ubuntu 24.04 host on 2026-10-28):
- An NVIDIA driver that supports CUDA 13.2 and uv.
- The Python 3.12 headers and a C compiler: Triton compiles a small C helper every time vLLM starts; without them every
vllm serve stops with
fatal error: Python.h: No such file or directory. On Debian/Ubuntu: sudo apt-get update && sudo apt-get install -y python3.12-dev build-essential.
- No CUDA toolkit is needed: the FlashInfer line below installs prebuilt kernels.
- Disk: about 20 GB for the venv, as much again for uv's cache (
uv cache clean frees it), plus 15 GB for the model.
EXL3 is not a native vLLM format. vllm-quantpack (ours, on PyPI) loads it through vLLM's plugin entry point: EXL3 linear and embedding layers through exllamav3's kernels, the vision tower, the MTP head and a fixed, faster int4 KV cache.
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \ --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match # FlashInfer kernelsW=exllamav3-1.6.5+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.6.5/${W/+/%2B}"uv pip install --python venv/bin/python /tmp/$W "vllm-quantpack>=0.1.4"venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Quick start (single RTX 3090, 24 GB)
The configuration of every table above (text only, no speculative decoding):
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwopus3.8-27B-Coder-Flash-EXL3-3.50bpw-24GB \ --served-model-name qwopus-coder \ --language-model-only \ --max-model-len 262144 \ --kv-cache-dtype int4_per_token_head \ --kv-cache-memory-bytes 6914897346 \ --max-num-seqs 8 \ --max-num-batched-tokens 8192 \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
On a card that also drives a display, lower --kv-cache-memory-bytes by what the desktop uses. On 48 GB GPUs raise it: the KV of this model is about 18 KB per token in int4 (16 of its 64 layers are full attention, with 4 KV heads of 256).
Using it with opencode (how we tested)
limit.context must match --max-model-len (262144 above).
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "qwopus-coder": { "name": "Qwopus3.8-27B-Coder-Flash EXL3 3.50 (local)", "tool_call": true, "temperature": true, "limit": { "context": 262144, "output": 32768 } } } } }, "agent": { "build": { "model": "local-vllm/qwopus-coder", "temperature": 1.0 }, "plan": { "model": "local-vllm/qwopus-coder", "temperature": 1.0 } }}
- Declare
"temperature": true in the model, otherwise opencode sends no temperature to a custom model.
- With
output: 32768 opencode compacts the history at about context − 32k tokens of input, before vLLM rejects the request.
Important notes
- Temperature 1.0, never 0. Lower temperatures loop on long answers (see Sampling and repetition loops); at temperature 0 the reasoning loops and no answer is written, as with the original BF16 model.
- Memory. The driver reserves about 453 MiB of a 24 GB card that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.free. Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization.
- Prefix caching matters for agents: opencode sends the same tools and system prompt in every request, and each turn repeats the whole conversation (a new 32k prompt takes about 30 s, a cached one about 2 s).
- Long prompts: the first turn of a 258k-token prompt takes about 7 minutes to prefill on a 3090; later turns reuse the cache.
- MTP and prefix caching (vLLM 0.31, vllm#53670). With MTP on, vLLM recomputes the last cached block (3,104 tokens) on every hit. Measured at 258k on one RTX 3090: a new conversation turn takes 23 s (18 s without MTP) and a new question over the same long document 20 s (16.5 s without MTP). Adding
--prefix-match-unit 16 --enable-mamba-shared-prefix-checkpoint (vLLM's opt-in fix; not in the Quick start, whose memory peak was measured without it) gave 9.6 s per turn and 14 s per question from the second one on. One case still misses the cache with MTP, with or without the fix: a prompt that is a shorter cut of a previous prompt ending in its last blocks (a 257,600-token cut of a 258,000-token prompt was prefilled again, about 440 s).
- Tool calls work with the base model's own template and
--tool-call-parser qwen3_coder; no extra template is needed.
- Hugging Face page warnings: the "Model size" shown by Hugging Face counts packed storage elements, not parameters; the Configuration Parsing Warning about
quantization_config.bits not being an integer is expected (EXL3 bits is an average).
Quantization recipe
Table with columns: Step, Setting| Step | Setting |
|---|
| Source | Jackrong/Qwopus3.8-27B-Coder-Flash, downloaded on 2026-11-01; chat template and tokenizer unchanged |
| Tool | exllamav3 1.6.5 convert.py, one RTX 3090, about 3 h |
| Linear layers | 3.50 bpw (-b 3.50) |
lm_head / vision tower / MTP head | 6 / 6 / 4 bits (-hb 6 -vb 6 -mb 4) |
| Token embeddings | BF16, unchanged |
| Calibration | 585 rows built from real opencode conversation prefixes of our sysadmin work plus agent steps written by Swift-1.5 (a Qwen3.8-27B fine-tune) in BF16 on our internal prompts; the prefixes are model-independent (same tokenizer). Not published, and no text is stored in the weights |
Integrity and validation
Artifact integrity
SHA256SUMS lists every file; the verifier compares the repo with our local copy byte for byte before publication.
- The safetensors files carry no
__metadata__; quantization_config.json holds only bit widths, tensor layout and the calibration size (rows and columns).
Quantization
- KL divergence against the BF16 model with exllamav3's
qbench on three traces (Results).
Runtime
- Load, capacity, 262k needle, tool calls with the real opencode request, agent and programming tasks, agents at once, loops and long prompts: see Results; engine errors during the runs: 0.
Scope: results apply to the stack (vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4), the RTX 3090 and the settings stated above; other setups are not tested.
Limitations
- Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine where you can.
- Will not load with stock vLLM, Transformers or SGLang: it needs exllamav3 and vllm-quantpack.
- Only one session of the full 262k context fits on a 24 GB card.
- Vision and MTP were measured with the short battery and one programming run only (vision 15/15, tools 30/30, programming 12/12; 23,602 MiB peak); the agent and loop results above are text-only without MTP.
- Not measured: long-context decode speed with vision on, reasoning problems at
xhigh, GPUs other than the RTX 3090. The calibration and our two traces are internal, so those KLD numbers cannot be reproduced outside; the public-trace KLD can.
Feedback welcome
Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
Credits & license