What is in this repository
Which one to use
All Qwen3.8-27B-pi variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):
Table with columns: Variant, Weights, Context on one RTX 3090, KLD vs BF16, Agent tasks, Decode, 1 request, Pick it for| Variant | Weights | Context on one RTX 3090 | KLD vs BF16 | Agent tasks | Decode, 1 request | Pick it for |
|---|
| GPTQ INT8 g64 | 15.75 GiB | 176,000 (text) · 131,072 with vision | 0.0182 | 29/30 | 96.5 tok/s | a more faithful GPTQ (KLD 0.0182) on stock vLLM (+ the same patch), with less context |
| EXL3 3.85 (this repo) | 13.62 GiB | 215,000 with vision | 0.0074 | 29/30 | 81.1 tok/s | the closest to BF16 (KLD 0.0074) with vision at 215,000; needs exllamav3 + our plugin |
| EXL3 3.70 (this repo) | 13.19 GiB | 226,000 with vision | 0.0093 | 28/30 | 80.1 tok/s | more context (226,000) and 4 sessions with vision, a little less faithful than 3.85 |
Variants / revisions
Table with columns: Revision, Linear layers, lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)| Revision | Linear layers | lm_head | Vision | MTP head | Embeddings | Size | KLD (ours / pub.) |
|---|
main | 3.85 bpw, SC allocation | 5 bits | 6 bits | 4 bits | INT8 per row | 13.62 GiB | 0.0074 / 0.0070 |
3.70bpw | 3.70 bpw, SC allocation | 5 bits | 6 bits | 4 bits | INT8 per row | 13.19 GiB | 0.0093 / 0.0085 |
Files and reproducibility
model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).
plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding; 0.2.0). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
LICENSE (Apache-2.0) and NOTICE (the pi model's notice, plus this quantization).
Results
Family comparison (Qwen3.8-27B-pi, same conditions)
Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (GPTQ: + the embedding patch / EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).
Table with columns: Variant, Weights, KLD mean / median (internal · public), Perplexity (BF16 1.5561 · pub. 1.3265), Needle (8 facts) near max. context, Tools: real opencode request, Agent tasks (10 × 3), Reasoning xhigh: correct · tokens, Vision| Variant | Weights | KLD mean / median (internal · public) | Perplexity (BF16 1.5561 · pub. 1.3265) | Needle (8 facts) near max. context | Tools: real opencode request | Agent tasks (10 × 3) | Reasoning xhigh: correct · tokens | Vision |
|---|
GPTQ W4A16 g64 + INT8 embeddings (main) | 15.75 GiB | 0.0182 / 0.0041 · pub. 0.0170 / 0.0007 | 1.5900 · pub. 1.3514 | 8/8 at 162,208 | 60/60 | 29/30 | 48/48 · 18,961 | 15/15 |
EXL3 3.85 bpw (main, this repo) | 13.62 GiB | 0.0074 / 0.0016 · pub. 0.0070 / 0.0003 | 1.5702 · pub. 1.3362 | 8/8 at 197,991 | 60/60 | 29/30 | 48/48 · 18,535 | 15/15 |
EXL3 3.70 bpw (3.70bpw, this repo) | 13.19 GiB | 0.0093 / 0.0020 · pub. 0.0085 / 0.0003 | 1.5706 · pub. 1.3386 | 8/8 at 208,158 | 60/60 | 28/30 | 48/48 · 20,618 | 15/15 |
Table with columns: Variant, Context on one RTX 3090, Sessions, Prefill near max. context, Decode, 1 / 2 / 8 requests (tok/s), MTP tokens per step, VRAM really free at peak| Variant | Context on one RTX 3090 | Sessions | Prefill near max. context | Decode, 1 / 2 / 8 requests (tok/s) | MTP tokens per step | VRAM really free at peak |
|---|
| GPTQ INT8 g64 | 176,000 (text) · 131,072 with vision | 2 | 680 tok/s | 96.5 / 178.6 / 184.7 | 3.12 | 196 MiB · vision config: 840 MiB ¹ |
| EXL3 3.85 | 215,000 with vision | 2 | 555 tok/s | 81.1 / 134.0 / 145.4 | 3.11 | 320 MiB |
| EXL3 3.70 | 226,000 with vision | 4 | 526 tok/s | 80.1 / 139.3 / 257.4 | 3.08 | 148 MiB |
How to read it:
- KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's
qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign.
- Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (
generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test.
- Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run.
- VRAM really free: lowest
memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB. ¹ While loading the weights, free memory dipped below 100 MiB for a moment (it started and served normally); on a GPU that also drives a display, leave a little more room.
- Older sections below keep their own date and setup; use this table to compare variants.
How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):
Table with columns: Variant, KV pool (tokens), Max sessions, 32k each, 64k each, 128k each, Full context each| Variant | KV pool (tokens) | Max sessions | 32k each | 64k each | 128k each | Full context each |
|---|
| GPTQ INT8 g64 | 178,816 | 2 | 2 | 2 | 1 | 1 × 176,000 |
| GPTQ INT8 g64 (vision config) | 132,423 | 2 | 2 | 2 | 1 | 1 × 131,072 |
| EXL3 3.85 | 216,433 | 2 | 2 | 2 | 1 | 1 × 215,000 |
| EXL3 3.70 | 231,757 | 4 | 4 | 3 | 1 | 1 × 226,000 |
Few users or many users. The Quick start configuration is tuned for 1–4 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.
Table with columns: Variant, Configuration, Decode by simultaneous requests (aggregate tok/s), N sessions at once, needle 8/8, Tools (5 × 3), VRAM really free| Variant | Configuration | Decode by simultaneous requests (aggregate tok/s) | N sessions at once, needle 8/8 | Tools (5 × 3) | VRAM really free |
|---|
| GPTQ INT8 g64 | Quick start (2 sessions, MTP 3) | 1: 96 · 2: 179 · 8: 185 | — | 60/60 | 196 MiB |
| GPTQ INT8 g64 | many users (8 sessions of up to 165,000 tokens, MTP 3) | 1: 97 · 2: 178 · 4: 322 · 8: 468 | 8/8 of 22k | 14/15 | 386 MiB |
| EXL3 3.85 | Quick start (2 sessions, MTP 3) | 1: 81 · 2: 134 · 8: 145 | — | 60/60 | 320 MiB |
| EXL3 3.85 | many users (8 sessions of up to 215,000 tokens, MTP 3) | 1: 80 · 2: 131 · 4: 249 · 8: 296 | 8/8 of 22k | 15/15 | 178 MiB |
| EXL3 3.70 | Quick start (4 sessions, MTP 3) | 1: 80 · 2: 139 · 8: 257 | — | 60/60 | 148 MiB |
| EXL3 3.70 | many users (8 sessions of up to 220,000 tokens, MTP 3) | 1: 79 · 2: 146 · 4: 259 · 8: 311 | 8/8 of 22k | 15/15 | 338 MiB |
Benchmarks (single RTX 3090 24 GB, 300 W)
Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.
Table with columns: Test, main 3.85 bpw (215,000), 3.70bpw (226,000)| Test | main 3.85 bpw (215,000) | 3.70bpw (226,000) |
|---|
| KV pool | 216,433 tokens (KV 8.42e9 B) | 227,439 tokens (KV 8.85e9 B) |
| 8 facts in one long prompt (greedy) | 8/8 at 61,675, 123,142 and 195,210 tokens (524 tok/s at 195k) | not measured |
| 2 simultaneous sessions | 2 × ≈94k, 8/8 in both, 698 tok/s aggregate | not measured |
| Decode, 1 request, t=0 / 0.6 | 76.8 / 75.5 tok/s (MTP: 70 % of drafts accepted) | not measured |
| Decode, 2 concurrent, t=0 / 0.6 | 135.2 / 125.5 tok/s | not measured |
| Images (4 synthetic tasks, 18 checks) | 4/4 tasks, 18/18; follow-up turn on the same image (prefix cache) 2.7 → 1.4 s | not measured |
| Real opencode request, tool calls (3 cases × 10 × 3 configurations) | 90/90 | not measured |
| Reasoning tokens, 6 problems × 8 (T 0.6): low · medium · xhigh | 16,933 (47/48) · 18,569 (48/48) · 17,070 (46/48) | not measured |
opencode run, 4 sysadmin tasks × 2 (xhigh, T 0.3) | 8/8 tasks, 4,279 reasoning tokens, 352 s, 41 requests | not measured |
| Peak / really free | 311 MiB really free at peak, no engine errors | not measured |
| Installed from this repo following this card (2026-10-10) | see note | see note |
For comparison, the base Qwen3.8-27B EXL3 3.85 reasoned 23,221 tokens at xhigh on the same problems (pi: 17,070, −26 %), with the same context and memory.
Sampling and repetition loops
Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 3.85 build of Qwen3.8-27B-pi (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.
Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Temperature | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| Loops / 30 | 9 | 4 | 9 | 10 | 4 | 2 | 1 | 0 |
| Answers that finished before 16,000 tokens / 30 | 10 | 12 | 10 | 7 | 9 | 11 | 8 | 13 |
From temperature 0.9 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 1.0.
Other models, same test and setup (loops / 30):
Table with columns: Model (build), 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Model (build) | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| Qwen3.8-27B · EXL3 3.85 | 12 | 12 | 7 | 8 | 1 | 1 | 1 | 0 |
| Qwopus3.8-27B-Flash-V2 · EXL3 3.85 | 3 | 1 | 2 | 1 | 2 | 0 | 0 | 0 |
| Swift 1.5 (Qwen3.8-27B) · EXL3 3.85 | 7 | 5 | 3 | 4 | 3 | 1 | 1 | 0 |
| Ornith-1.5-9B · EXL3 6.00 (16 GB) | 6 | 2 | 3 | 1 | 1 | 0 | 0 | 0 |
| NeoHorse-1-9B · EXL3 6.00 (16 GB) | 6 | 3 | 3 | 1 | 0 | 0 | 0 | 0 |
| Qwen3.6-35B-A3B · EXL3 4.00 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
| Ornith-1.5-35B-A3B · EXL3 4.00 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Qwopus3.6-35B-A3B-Coder · EXL3 4.00 | 9 | 13 | 12 | 6 | 11 | 6 | 5 | 4 |
| Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM | 24 | 24 | 21 | 21 | 17 | 8 | 0 | 0 |
| Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM | — | — | 13 | 7 | 5 | 2 | 0 | 0 |
Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)
EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:
- Continuum-AI-Corp/OrcaSAQ2-kernel (commit
7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear
layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed).
qwen38_exl3_vision 0.4.1 (ours; 0.4.1 is a maintenance release; 0.4.0 added multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision
tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's
fused kernel, and the Qwen3.8-Flash-Next model.
Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels
are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match,
point CUDA_HOME at a CUDA 13.0 toolkit.
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto # torch 2.13.0W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Quick start (single RTX 3090, 24 GB)
main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP):
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwen3.8-27B-pi-EXL3-3.85bpw-24GB \ --served-model-name qwen3.8-27b-pi \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 215000 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 8420000000 \ --max-num-batched-tokens 1024 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
3.70bpw branch, 226k context: the same command with --revision 3.70bpw --max-model-len 226000 --kv-cache-memory-bytes 9000000000 --max-num-seqs 4 (4 sessions; validated in Family comparison).
For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 215,000 tokens, MTP 3. Aggregate decode 296 tok/s with 8 requests (Quick start: 145 with 8); a single request runs at 80 tok/s (81 with the Quick start); 8 of 8 simultaneous sessions returned all 8 needle facts; 178 MiB of VRAM really free. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.
Using it with opencode (how we tested)
This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was
measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks;
context, speed, memory and image tests were run directly against the vLLM API.
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "qwen3.8-27b-pi": { "name": "Qwen3.8-27B-pi EXL3 (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 215000, "output": 22000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 1.0 }, "plan": { "model": "local-vllm/qwen3.8-27b-pi", "variant": "xhigh", "temperature": 1.0 } }}
-
Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and in our agent tests (10 real sysadmin tasks × 3 runs with opencode run) 1.0 scored the same as 0.3 within one task: 29/30 at 1.0 and 30/30 at 0.3 for Qwen3.8-27B, -pi and Swift 1.5 27B; Qwopus3.8, served with its chat_template-xml-tools.jinja, 28/30 at 1.0 and 26/30 at 0.3.
-
limit.context = --max-model-len (226000 for the 3.70bpw branch); with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
-
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests).
-
Without attachment / modalities opencode will not send images; drop them if you serve with --language-model-only.
-
Start opencode inside a project directory (it takes a git snapshot before each step); for opencode run from scripts, close stdin (</dev/null).
Important notes
- Memory. The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.used,memory.free --format=csv, not as 24,576 − used. Size the KV cache with --kv-cache-memory-bytes, never with
--gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). --max-num-batched-tokens 1024 and
expandable_segments lower the peak by ≈350 MiB at no measurable cost. With MTP, KV capacity grows in block-sized steps: check the GPU KV cache size
line (8,420,000,000 B → 216,433 tokens; 8,850,000,000 B → 227,439) and keep --max-model-len below it.
- Measured peaks (2 sessions, vision): 3.85 bpw 311 MiB really free (719 MiB at rest); 3.70 bpw see the benchmark table. Leave the GPU for the LLM: a desktop session on the same GPU would eat this margin.
- Hugging Face page warnings: the "Model size" shown by Hugging Face (≈8B params) counts the packed storage elements of the safetensors files
(EXL3 packs ≈4 weights into each
int16), not parameters: this is the 27B model at ≈3.85 bits per weight. The Configuration Parsing Warning
"quantization_config.bits must be an integer" is also expected: for EXL3 bits is the average bits per weight of a per-tensor allocation
(the per-tensor values are in quantization_config.json); exllamav3 and the OrcaSAQ2 plugin read it as a float.
VLLM_USE_FLASHINFER_SAMPLER=0 is recommended by the OrcaSAQ2 plugin (flashinfer's JIT sampler is fragile; torch's is equivalent).
- Use
--tool-call-parser qwen3_coder (Qwen3.8 XML tool calls) and --reasoning-parser qwen3. reasoning_effort accepts only xhigh, medium and low
(high returns HTTP 400); they are prompt instructions, not limits. For a hard cap, thinking_token_budget was validated with vLLM 0.29 on the W4A16 version
(not re-checked with this stack).
- On Ampere (sm86) the fp8 KV cache works with the FlashInfer attention backend (the default).
- Why it is slower than W4A16 on Ampere (per-layer micro-benchmarks on the 3090): with 1 row the EXL3 kernel matches Marlin, but with the 4–8 rows of an
MTP verification step decoding the trellis becomes compute-bound and costs ≈6 ms more per step; the plugin path (BF16↔FP16 conversions, fused layers run
piecewise) adds ≈2.6 ms, and a higher-bit
lm_head ≈1–2 ms. Ada / Blackwell GPUs should lose less. Prefill is ≈8 % slower.
- Stability (our machine): our RTX 3090 hung twice at its factory 370 W limit with very long prompts (once inside an exllamav3 kernel, once with an
unrelated model); after setting a 350 W (now 300 W) power limit and a more aggressive fan curve, none of the tests here (up to 245k-token prompts) hung again. If you
see Xid errors in
dmesg, try a lower power limit first.
Quantization recipe
Table with columns: Step, What we did| Step | What we did |
|---|
| Calibration trace | Generated by Qwen3.8-27B-pi itself (reasoning_effort: xhigh, MTP enabled) with exllamav3's sc_trace on 196 prompts: 157 from exllamav3's own sc_trace set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published); 196 sequences × 2048 tokens used for conversion |
| Reference and sensitivity | sc_rfn_probe against a 2.00 bpw reference, sc_measure (per-tensor sensitivity on the trace) |
| Bit allocation | sc_optimize at the target rate with a 5-bit lm_head → per-tensor recipe (quantization_config.json) |
| Conversion | exllamav3 1.5.1 convert.py with the recipe and the calibration trace, vision 6 bits, MTP 4 bits; codebook mul1, output scales always |
| Embeddings | embed_tokens re-encoded to INT8 per row (absmax / 127, BF16 scales) in the OrcaSAQ2 plugin's format; script included |
- Tokenizer, chat template and generation config unchanged from bytkim/Qwen3.8-27B-pi (XML tool calls,
qwen3_coder). The dflash2/ companion of the source repo is not included.
Integrity and validation
What was checked for this checkpoint, how, and where the numbers are (measured on 2026-10-14; the method and every figure are in Family comparison).
Artifact integrity
- Every weight file is checked against our local copy with SHA-256 before publication (
SHA256SUMS in this repo).
- The measurements used the weights downloaded from this repo at commit
1f86a4016988; later commits change only metadata, documentation or the plugin, never the .safetensors files.
- vLLM loads every tensor listed in
model.safetensors.index.json on start-up (a missing tensor or a wrong shape would stop it): checked from a fresh download.
- After the metadata update of 2026-10-14 (
quantization_config.json, plugin 0.3.1, embed_int8.py): downloaded again from this repo and re-tested: loads; vision 15/15 needle 64k 8/8; tool calls 15/15; engine errors 0.
Quantization
- Per-tensor bit allocation is recorded by exllamav3 in
quantization_config.json; its tensor_storage entry for embed_tokens describes the INT8 qweight + BF16 scales actually stored (updated on 2026-10-14; the plugin only reads the .trellis entries).
- The INT8 embedding conversion (
scripts/embed_int8.py) is reproducible: re-running it on the source checkpoint gives the published qweight and scales bit for bit (checked on Ornith-1.5-9B 6.00 bpw); its reported error now uses the stored BF16 scales (0.00806 vs 0.00803 with FP32 scales).
- Fidelity: KL divergence against the BF16 Qwen3.8-27B-pi with exllamav3's
qbench on two traces (ours and turboderp's public one).
Chat template
- The published template renders correctly in 40 edge cases (empty and very long system prompt, no tools, two tool calls in one message, nested JSON and multi-line arguments, images with tools, two images, earlier reasoning, multi-turn with several tool results) × the reasoning settings (
enable_thinking: false, reasoning_effort low / xhigh): all pass (prueba_plantillas.py). With Transformers directly, pass tool-call arguments as dicts (vLLM converts JSON strings itself).
Runtime
- Load, long-context needle, vision, tool calls with the real opencode request, 10 agent tasks × 3 runs, concurrency and VRAM: see Family comparison; engine errors during the run: 0.
- On a desktop RTX 3090 that also drives the display (300 W): Quick start at 215,000 tokens: 317 MiB really free.
Scope: results apply to the revision, software stack (vLLM 0.31.0 + exllamav3 1.5.1 + plugins), hardware and settings stated above; other setups are not tested.
Limitations
- Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with
sudo, and give the agent a non-privileged account and a test machine where you can.
- Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images,
short reasoning problems and agent tasks.
- The full 262,144 context does not fit at this rate on 24 GB with MTP and vision.
- Decode is ≈20 % slower than the W4A16 sibling on Ampere; the OrcaSAQ2 plugin is young and describes itself as a correctness path.
- The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.
Feedback welcome
Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver,
vLLM / exllamav3 / plugin versions, the full command line and the result (speed, context, or the error message).
Credits & license
Credits & acknowledgements
License
Apache-2.0, same as the pi model and Qwen3.8-27B; the pi model's NOTICE is kept in NOTICE. The OrcaSAQ2 plugin and exllamav3 are not
redistributed here; install them from their repositories under their own licenses.