Which one should you use?
Both builds come from the same model, the same calibration data and the same exllamav3 version; only the bit rate differs.
Table with columns: 2.25 bpw (this repo), 3.50 bpw | 2.25 bpw (this repo) | 3.50 bpw |
|---|
| Weights | 11.19 GB | 14.99 GB |
| Full 262k sessions at once on one RTX 3090 | 2 | 1 |
| KV pool on one RTX 3090 | 586,300 tokens | 377,712 tokens |
| KLD vs BF16 (opencode / public / sysadmin) | 0.089 / 0.080 / 0.109 | 0.013 / 0.010 / 0.015 |
| Agent tasks (3 × 10) | 25/30 | 30/30 |
| Tool calls (30) | 29 | 30 |
| Programming tasks (3 × 12) | 35/36 | 36/36 |
| Loops at temperature 1.0 (30 long answers) | 8 | 0 |
| Decode near 260k, one request | 27.2 tok/s | 25.4 tok/s |
| Vision and MTP measured | no | yes |
Pick this 2.25 build when two people (or two agents) must work with very long contexts on the same 24 GB card at the same time, and an occasional failed task or looping answer is acceptable because a human reviews the result and can retry.
Pick the 3.50 build when one long session is enough, or when correctness matters more than concurrency: it passed every agent task, did not loop at temperature 1.0 and is about 7 times closer to BF16. If you have two GPUs, two copies of the 3.50 build are better than one 2.25.
Shorter sessions share the pool in both builds: by vLLM's count this build holds 4 sessions of up to 146k or 8 of up to 73k tokens at once (not load-tested at those sizes).
What is in this repository
Table with columns: File, What it is| File | What it is |
|---|
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json | EXL3 weights (11.19 GB) and their index |
quantization_config.json | exllamav3's per-tensor storage record (bits per tensor; calibration size only, no text) |
config.json, generation_config.json | model config; sampling defaults of the base model |
chat_template.jinja, tokenizer.json, tokenizer_config.json | chat template and tokenizer of the base model, unchanged |
preprocessor_config.json, processor_config.json, video_preprocessor_config.json | image/video processor of the base model |
SHA256SUMS | SHA-256 of every file above |
Not included: the BF16 reference, the calibration data (internal), the plugin (install it from PyPI, see below).
Results
All tables: one RTX 3090 24 GB (no other use), vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4, --max-model-len 262144, --kv-cache-dtype int4_per_token_head, text only, the Quick start configuration, measured on 2026-11-02 unless stated otherwise.
Quantization quality (exllamav3 qbench, against BF16)
Table with columns: Trace, KLD mean, KLD median, KLD p90, Perplexity (BF16)| Trace | KLD mean | KLD median | KLD p90 | Perplexity (BF16) |
|---|
| opencode agent trace (ours) | 0.0888 | 0.0290 | 0.245 | 2.7367 (2.5982) |
| sysadmin trace (ours) | 0.1087 | 0.0351 | 0.303 | 1.8631 (1.6332) |
public (qbench_prompts_gen.json, turboderp) | 0.0798 | 0.0099 | 0.244 | 1.5668 (1.4128) |
How to read it: the median is low, so most tokens are predicted almost as BF16 does, but the p90 is about 7 times that of the 3.50 build: one token in ten differs clearly from BF16. Those tokens are where the extra loops and failed tasks come from.
Capacity on one RTX 3090 (24 GB)
Table with columns: --kv-cache-memory-bytes, KV pool reported by vLLM, Real concurrent sessions of 235,929 tokens, Peak used, Really free at peak--kv-cache-memory-bytes | KV pool reported by vLLM | Real concurrent sessions of 235,929 tokens | Peak used | Really free at peak |
|---|
| 10705205985 | 586,300 tokens | 2 (8/8 facts found in each; a third waits) | 22,320 MiB | 1,796 MiB |
The token embeddings are stored in BF16 (2.54 GB); the KV of this model is about 18 KB per token in int4. Aggregate prefill with 1 / 2 sessions of 235k: 636 / 1,250 tok/s.
A 16 GB card does not fit the full context: with a 5.0e9-byte KV pool (1.04 × 262k) the peak was 16.5 GB.
Test battery
Table with columns: Test, Result| Test | Result |
|---|
| Tool calls, battery | 29/30 valid and correct, 0 errors |
Agent tasks, opencode run, 10 sysadmin tasks × 3 runs, temperature 1.0 | 8/10, 9/10, 8/10 (479-725 s per run) |
| Programming tasks (12 × 3) | 11/12, 12/12, 12/12 (2,107-2,542 s per run) |
| Needle, 8 facts in one prompt | 64k 8/8 · 128k 8/8 · 241k 8/8 (second machine: 64k · 128k · 245k · 258k, 8/8 each) |
| Prefill / decode near 15k · 64k · 128k · 260k | 1,134 / 48.7 · 980 / 41.8 · 822 / 35.5 · 618 / 27.2 tok/s |
| Decode with 1 / 2 / 4 / 8 short requests | 50 / 72 / 132 / 244 tok/s in total |
| Agents at once, tasks passed (runs that reached the 600 s limit) | 4: 8/8 (1) · 8: 14/16 (7) · 16: 27/32 (14) |
| Time to first token of a new 32k prompt (no cache / cached), with 4 · 8 · 16 agents running | 29.9 / 1.6 s · 30.2 / 2.0 s · 33.9 / 7.7 s |
With sampling on, the same checkpoint varies by about ±2 of 30 between runs; the gap with the 3.50 build in agent tasks (25 against 30) is larger than that.
Comparison: the five Qwopus3.8-Coder-Flash builds we tested
Same model, two bit rates and three calibration choices; same card, settings and tests as above.
Table with columns: Build, KLD opencode / public / sysadmin, Real 236k sessions, Programming, Loops /30 at T 0.6 / 0.8 / 1.0, Tools (30), Agent (3 × 10)| Build | KLD opencode / public / sysadmin | Real 236k sessions | Programming | Loops /30 at T 0.6 / 0.8 / 1.0 | Tools (30) | Agent (3 × 10) |
|---|
| 2.25, no calibration data | 0.0964 / 0.0860 / 0.1205 | 2 | 35/36 | 24 / 16 / 21 | 27 | 27/30 |
| 2.25, calibration as this repo (this repo) | 0.0888 / 0.0798 / 0.1087 | 2 | 35/36 | 14 / 13 / 8 | 29 | 25/30 |
| 2.25, steps written by Qwopus | 0.0869 / 0.0810 / 0.1081 | 2 | 36/36 | 26 / 16 / 10 | 30 | 27/30 |
| 3.50, calibration as this repo (repo) | 0.0127 / 0.0104 / 0.0148 | 1 | 36/36 | 12 / 6 / 0 | 30 | 30/30 |
| 3.50, steps written by Qwopus | 0.0126 / 0.0105 / 0.0152 | 1 | 35/36 | 18 / 7 / 2 | 30 | 28/30 |
Why this 2.25 and not the others: the three 2.25 builds are within about 10 % in KL divergence, and the agent and tool differences between them are inside the run-to-run noise, but this one loops clearly less (8 of 30 at temperature 1.0, against 21 and 10). At 2.25 bpw the calibration data lowers the KL divergence by 8-10 % against no calibration.
Sampling and repetition loops
5 long Spanish sysadmin questions × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated word 8-grams.
Table with columns: Temperature, 0.6, 0.8, 1.0| Temperature | 0.6 | 0.8 | 1.0 |
|---|
| Loops / 30 | 14 | 13 | 8 |
| Answers that finished before 16,000 tokens / 30 | 19 | 19 | 22 |
| Answers with no text (only reasoning) / 30 | 13 | 13 | 10 |
| Mean tokens per answer | 9,767 | 9,885 | 10,039 |
Use temperature 1.0 (the default in generation_config.json); lower temperatures loop more, and temperature 0 does not work with this model (see the warning at the top). Even at 1.0 about a quarter of long answers loop: in an agent this shows up as a turn that keeps reasoning until max_tokens, so keep a limit and let the agent retry.
Known model behaviour and discarded options
- Speed: a dense 27B; one request decodes at 49 tok/s near 15k tokens and 27 tok/s near 260k on a 3090, so long agent tasks take minutes. With 8 or 16 agents at once on one card many tasks reach a 10-minute limit.
- Temperature 0 loops: greedy decoding loops inside the reasoning on long answers; we measured it in BF16 and in the 3.50 build (see the warning at the top).
- Not measured with this build: vision and MTP speculative decoding. The vision tower (6 bits) and the MTP head (4 bits) are in the files, but every test above used
--language-model-only and no speculative decoding. The 3.50 build, with the same vision tower and MTP head bits, passed vision 15/15 with MTP 3.
Requirements & install (vLLM 0.31 + exllamav3 1.6.5 + vllm-quantpack 0.1.4)
Before you install (the same stack as our other cards, checked on a clean Ubuntu 24.04 host on 2026-10-28):
- An NVIDIA driver that supports CUDA 13.2 and uv.
- The Python 3.12 headers and a C compiler: Triton compiles a small C helper every time vLLM starts; without them every
vllm serve stops with
fatal error: Python.h: No such file or directory. On Debian/Ubuntu: sudo apt-get update && sudo apt-get install -y python3.12-dev build-essential.
- No CUDA toolkit is needed: the FlashInfer line below installs prebuilt kernels.
- Disk: about 20 GB for the venv, as much again for uv's cache (
uv cache clean frees it), plus 12 GB for the model.
EXL3 is not a native vLLM format. vllm-quantpack (ours, on PyPI) loads it through vLLM's plugin entry point: EXL3 linear and embedding layers through exllamav3's kernels, the vision tower, the MTP head and a fixed, faster int4 KV cache.
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \ --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match # FlashInfer kernelsW=exllamav3-1.6.5+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.6.5/${W/+/%2B}"uv pip install --python venv/bin/python /tmp/$W "vllm-quantpack>=0.1.4"venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Quick start (single RTX 3090, 24 GB)
The configuration of every table above (text only, no speculative decoding, two full-context sessions):
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwopus3.8-27B-Coder-Flash-EXL3-2.25bpw-24GB \ --served-model-name qwopus-coder \ --language-model-only \ --max-model-len 262144 \ --kv-cache-dtype int4_per_token_head \ --kv-cache-memory-bytes 10705205985 \ --max-num-seqs 8 \ --max-num-batched-tokens 8192 \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
On a card that also drives a display, lower --kv-cache-memory-bytes by what the desktop uses (each GB less is about 55k tokens less in the pool; below about 9.6e9 bytes the second full-context session no longer fits).
Using it with opencode (how we tested)
limit.context must match --max-model-len (262144 above).
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "qwopus-coder": { "name": "Qwopus3.8-27B-Coder-Flash EXL3 2.25 (local)", "tool_call": true, "temperature": true, "limit": { "context": 262144, "output": 32768 } } } } }, "agent": { "build": { "model": "local-vllm/qwopus-coder", "temperature": 1.0 }, "plan": { "model": "local-vllm/qwopus-coder", "temperature": 1.0 } }}
- Declare
"temperature": true in the model, otherwise opencode sends no temperature to a custom model.
- With
output: 32768 opencode compacts the history at about context − 32k tokens of input, before vLLM rejects the request.
Important notes
- Temperature 1.0, never 0. Lower temperatures loop more on long answers; at temperature 0 the reasoning loops and no answer is written, as with the original BF16 model.
- Review the results. With 25/30 agent tasks and 8/30 looping long answers, this build makes mistakes the 3.50 build does not; keep a human in the loop.
- Memory. The driver reserves about 453 MiB of a 24 GB card that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.free. Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization.
- Prefix caching matters for agents: opencode sends the same tools and system prompt in every request, and each turn repeats the whole conversation (a new 32k prompt takes about 30 s, a cached one about 2 s).
- Long prompts: the first turn of a 258k-token prompt takes about 7 minutes to prefill on a 3090; two such prompts at once share the card (1,250 tok/s in total).
- Tool calls work with the base model's own template and
--tool-call-parser qwen3_coder; no extra template is needed.
- Hugging Face page warnings: the "Model size" shown by Hugging Face counts packed storage elements, not parameters; the Configuration Parsing Warning about
quantization_config.bits not being an integer is expected (EXL3 bits is an average).
Quantization recipe
Table with columns: Step, Setting| Step | Setting |
|---|
| Source | Jackrong/Qwopus3.8-27B-Coder-Flash, downloaded on 2026-11-01; chat template and tokenizer unchanged |
| Tool | exllamav3 1.6.5 convert.py, one RTX 3090 |
| Linear layers | 2.25 bpw (-b 2.25) |
lm_head / vision tower / MTP head | 6 / 6 / 4 bits (-hb 6 -vb 6 -mb 4) |
| Token embeddings | BF16, unchanged |
| Calibration | 585 rows built from real opencode conversation prefixes of our sysadmin work plus agent steps written by Swift-1.5 (a Qwen3.8-27B fine-tune) in BF16 on our internal prompts; the prefixes are model-independent (same tokenizer). Not published, and no text is stored in the weights |
Integrity and validation
Artifact integrity
SHA256SUMS lists every file; the verifier compares the repo with our local copy byte for byte before publication.
- The safetensors files carry no
__metadata__; quantization_config.json holds only bit widths, tensor layout and the calibration size (rows and columns).
Quantization
- KL divergence against the BF16 model with exllamav3's
qbench on three traces (Results).
Runtime
- Load, capacity with two full-context sessions, 262k needle, tool calls, agent and programming tasks, agents at once, loops and long prompts: see Results; engine errors during the runs: 0.
Scope: results apply to the stack (vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4), the RTX 3090 and the settings stated above; other setups are not tested.
Limitations
- Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine where you can.
- Lower quality than the 3.50 build: about 7 times its KL divergence, 25/30 agent tasks against 30/30, and loops on long answers even at temperature 1.0.
- Will not load with stock vLLM, Transformers or SGLang: it needs exllamav3 and vllm-quantpack.
- Not measured: vision, MTP speculative decoding, the real opencode request replay, reasoning problems at
xhigh, GPUs other than the RTX 3090. The calibration and our two traces are internal, so those KLD numbers cannot be reproduced outside; the public-trace KLD can.
Feedback welcome
Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
Credits & license