What is in this repository
Which one to use
Table with columns: GPU, Build, Context per session, Real concurrent sessions of 262k, KV cache, Pick it for| GPU | Build | Context per session | Real concurrent sessions of 262k | KV cache | Pick it for |
|---|
| 24 GB (RTX 3090) | this repo (4.0 bpw) | 262,144 with vision | 1 (pool 462,275 tokens) | int4 | ZwZ2's visual reasoning on one 24 GB card, full context |
| 16 GB | 2.25 bpw | 262,144 with vision | 1 (pool 284,694 tokens) | int4 | this build does not fit (15.03 GB of weights); the 2.25 does, at about ten times the KL divergence |
Files and reproducibility
Table with columns: File, What it is| File | What it is |
|---|
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json | EXL3 weights (15.03 GB) with INT8 token embeddings, and their index |
quantization_config.json | exllamav3's per-tensor storage record (bits per tensor; calibration size only, no text) |
config.json, generation_config.json | model config (quantization_config: exl3, 4.0 bits, head 6, vision 6); sampling defaults of the source model (temperature 1.0, top_p 0.95, top_k 20, enable_thinking: false) |
chat_template.jinja, tokenizer.json, tokenizer_config.json | chat template and tokenizer of the source model, unchanged |
processor_config.json | image/video processor of the source model, unchanged |
preprocessor_config.json, video_preprocessor_config.json | derived from processor_config.json (exllamav3 needs them; image_processor_type: Qwen2VLImageProcessorFast) |
scripts/embed_int8.py | re-encodes embed_tokens to INT8 per row (the step applied after the exllamav3 conversion) |
LICENSE | Apache-2.0 |
SHA256SUMS | SHA-256 of every file above |
Not included: the BF16 reference, the calibration data (internal), the plugin (install it from PyPI, see below), the source repository's images.
Results
All tables: one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4, the Quick start configuration (vision on, --max-model-len 262144, --kv-cache-dtype int4_per_token_head), measured on 2026-11-04 unless stated otherwise. KL divergence: exllamav3's qbench on an RTX PRO 6000 Blackwell.
Quantization quality (exllamav3 qbench, against BF16)
Table with columns: Trace, KLD mean, KLD median, KLD p90, Perplexity (BF16)| Trace | KLD mean | KLD median | KLD p90 | Perplexity (BF16) |
|---|
| opencode agent trace (ours) | 0.0090 | 0.0020 | 0.015 | 3.1909 (3.1931) |
| sysadmin trace (ours) | 0.0061 | 0.0020 | 0.016 | 2.7260 (2.7112) |
public (qbench_prompts_gen.json, turboderp) | 0.0066 | 0.0008 | 0.019 | 1.8500 (1.8450) |
The traces were written by other Qwen3.x models (same tokenizer) in thinking mode; ZwZ2 answers without reasoning by default, so these numbers measure how faithfully the quantized weights follow BF16 on that text, not ZwZ2's own style.
Capacity on one RTX 3090 (24 GB)
Table with columns: --kv-cache-memory-bytes, KV pool reported by vLLM, Real concurrent sessions of 262,144 tokens, Peak used, Really free at peak--kv-cache-memory-bytes | KV pool reported by vLLM | Real concurrent sessions of 262,144 tokens | Peak used | Really free at peak |
|---|
| 8464461987 | 462,275 tokens | 1 (1.76 by vLLM's count; not load-tested with 2) | 23,722 MiB | about 400 MiB |
Shorter sessions share the pool: with --max-num-seqs 4, four sessions of up to 115k fit at once by vLLM's count (not load-tested at that size).
Test battery
Table with columns: Test, Result| Test | Result |
|---|
| Smoke (CJK, loops, answers cut by the limit; three long Spanish answers) | OK: CJK 0, no loops, all three finished (1,556 / 2,054 / 2,051 tokens) |
| Tool calls with the real opencode request, battery | 30/30 valid and correct, 0 errors |
| Tool calls with the real opencode request, 3 cases × 20 (on an RTX PRO 6000, same settings at 32k) | 20/20, 20/20, 20/20 = 60/60, 0 errors |
| Vision (terminal screenshot, chart, dense text and other synthetic images) | 15/15 |
| Programming tasks (12, two runs) | 12/12 (413 s), 11/12 (421 s) |
| Decode with 1 / 2 / 4 / 8 short requests | 43.3 / 73.1 / 133.8 / 133.9 tok/s |
| Prefill of a 258,000-token prompt / decode of 300 tokens after it (cached) | 439 s / 16.9 tok/s |
| Needle, 8 facts in one prompt at 64k / 128k / 241k tokens (on an RTX PRO 6000, same settings) | 8/8, 8/8, 8/8 |
| Engine errors, out-of-memory in the vLLM log | 0, 0 |
Calibration: two choices compared
Table with columns: Build, KLD opencode / sysadmin / public, Tools (30), Vision (15), Programming (12)| Build | KLD opencode / sysadmin / public | Tools (30) | Vision (15) | Programming (12) |
|---|
| calibration from our sysadmin prefixes + steps written by Swift 1.5 (this repo) | 0.0090 / 0.0061 / 0.0066 | 30 | 15 | 12 |
| same prefixes + steps written by ZwZ2 itself (not published) | 0.0089 / 0.0061 / 0.0067 | 30 | 14 | 12 |
The two are equal within noise; this one was chosen for the vision result. (First run of both, before the enable_thinking flag was added to the configuration: same KV pool and memory peak.)
Sampling and repetition loops
5 long Spanish sysadmin questions × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated word 8-grams. Served with the Quick start settings at a 32k context and 8 sessions, on an RTX A40.
Table with columns: Temperature, 0.6, 0.8, 1.0| Temperature | 0.6 | 0.8 | 1.0 |
|---|
| Loops / 30 | 0 | 0 | 1 |
| Answers that finished before 16,000 tokens / 30 | 30 | 30 | 30 |
| Empty answers / 30 | 0 | 0 | 0 |
Answers average about 2,100 tokens (reasoning off). Loops are rare at every temperature tested; this repo keeps the source model's default (temperature 1.0, top_p 0.95, top_k 20).
Known model behaviour and discarded options
- Reasoning off by default: see the warning at the top. With
enable_thinking: true ZwZ2 reasons in English before answering, and its answers are much longer (3,916 against 1,937 tokens for the same Spanish sysadmin question in our check).
<answer> tags: when there is no system message, ZwZ2's template adds one that asks for the final answer inside <answer>...</answer>; the answers end with </answer>. A custom system message (opencode sends one) replaces that instruction.
- Speed: a dense 27B without speculative decoding; one request decodes at about 43 tok/s on a 3090.
Requirements & install (vLLM 0.31 + exllamav3 1.6.5 + vllm-quantpack 0.1.4)
Before you install (the same stack as our other cards, checked on a clean Ubuntu 24.04 host on 2026-10-28):
- An NVIDIA driver that supports CUDA 13.2 and uv.
- The Python 3.12 headers and a C compiler: Triton compiles a small C helper every time vLLM starts; without them every
vllm serve stops with
fatal error: Python.h: No such file or directory. On Debian/Ubuntu: sudo apt-get update && sudo apt-get install -y python3.12-dev build-essential.
- No CUDA toolkit is needed: the FlashInfer line below installs prebuilt kernels.
- Disk: about 20 GB for the venv, as much again for uv's cache (
uv cache clean frees it), plus 16 GB for the model.
EXL3 is not a native vLLM format. vllm-quantpack (ours, on PyPI) loads it through vLLM's plugin entry point: EXL3 linear and embedding layers through exllamav3's kernels, the INT8 embeddings, the vision tower and a fixed, faster int4 KV cache.
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \ --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match # FlashInfer kernelsW=exllamav3-1.6.5+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.6.5/${W/+/%2B}"uv pip install --python venv/bin/python /tmp/$W "vllm-quantpack>=0.1.4"venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Quick start (single RTX 3090, 24 GB)
The configuration of every table above (vision on, full context):
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:Truevllm serve JRamirez-UAB/ZwZ2-27B-EXL3-4.0bpw-24GB \ --served-model-name zwz2 \ --max-model-len 262144 \ --kv-cache-dtype int4_per_token_head \ --kv-cache-memory-bytes 8464461987 \ --max-num-seqs 4 \ --max-num-batched-tokens 2048 \ --enable-prefix-caching \ --default-chat-template-kwargs '{"enable_thinking": false}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
On a card that also drives a display, lower --kv-cache-memory-bytes by what the desktop uses. On 48 GB GPUs raise it.
Using it with opencode (how we tested)
limit.context must match --max-model-len (262144 above).
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "zwz2": { "name": "ZwZ2-27B EXL3 4.0 (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 262144, "output": 32768 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "think": { "chat_template_kwargs": { "enable_thinking": true } } } } } } }, "agent": { "build": { "model": "local-vllm/zwz2", "temperature": 1.0 }, "plan": { "model": "local-vllm/zwz2", "temperature": 1.0 } }}
- Declare
"temperature": true in the model, otherwise opencode sends no temperature to a custom model.
- Without
attachment / modalities opencode will not send images.
- With
output: 32768 opencode compacts the history at about context − 32k tokens of input, before vLLM rejects the request.
Important notes
- Memory. The driver reserves about 453 MiB of a 24 GB card that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.free. Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization.
- Prefix caching matters for agents: opencode sends the same tools and system prompt in every request, and each turn repeats the whole conversation.
- Tool calls work with the source model's own template and
--tool-call-parser qwen3_coder.
- Hugging Face page warnings: the "Model size" shown by Hugging Face counts packed storage elements, not parameters; the Configuration Parsing Warning about
quantization_config.bits not being an integer may appear (EXL3 bits is an average).
Quantization recipe
Table with columns: Step, Setting| Step | Setting |
|---|
| Source | inclusionAI/ZwZ2-27B revision 6decedf05c, downloaded on 2026-11-03; chat template and tokenizer unchanged |
| Tool | exllamav3 1.6.5 convert.py, one RTX A40 |
| Linear layers | 4.0 bpw, uniform (-b 4.0) |
lm_head / vision tower | 6 / 6 bits (-hb 6 -vb 6); no MTP head in the source |
| Token embeddings | re-encoded to INT8 per row (absmax / 127, BF16 scales) with scripts/embed_int8.py; relative error 0.0083; about 1.27 GB less VRAM |
| Calibration | 585 rows built from real opencode conversation prefixes of our sysadmin work plus agent steps written by Swift 1.5 (a Qwen3.8-27B fine-tune) in BF16 on our internal prompts; same tokenizer. Not published, and no text is stored in the weights |
Integrity and validation
Artifact integrity
SHA256SUMS lists every file; the verifier compares the repo with our local copy byte for byte before publication.
- The safetensors files carry no
__metadata__; quantization_config.json holds only bit widths, tensor layout and the calibration size (rows and columns).
Quantization
- KL divergence against the BF16 model with exllamav3's
qbench on three traces (Results).
Runtime
- Load, capacity at 262,144 with vision, needle up to 241k, tool calls with the real opencode request, vision, programming tasks and loops: see Results; engine errors during the runs: 0.
Scope: results apply to the stack (vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4), the RTX 3090 and the settings stated above; other setups are not tested.
Limitations
- Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine where you can.
- Will not load with stock vLLM, Transformers or SGLang: it needs exllamav3 and vllm-quantpack.
- Only one session of the full 262k context fits on a 24 GB card.
- ZwZ2 is a visual-reasoning model; our tests are sysadmin and generic vision checks (synthetic images: terminal screenshot, chart, dense text), not the authors' 25 benchmarks. Their scores are for the BF16 model.
- Not measured: agent tasks with
opencode run, reasoning problems, refusals, GPUs other than the RTX 3090. The calibration and our two traces are internal, so those KLD numbers cannot be reproduced outside; the public-trace KLD can.
Feedback welcome
Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
Credits & license