What is in this repository
Variants / revisions
Table with columns: Revision, Linear layers (achieved), lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)| Revision | Linear layers (achieved) | lm_head | Vision | MTP head | Embeddings | Size | KLD (ours / pub.) |
|---|
main | 3.85 bpw (3.866), SC allocation | 5 bits | 6 bits | 4 bits | INT8 per row | 13.64 GiB | 0.0073 / 0.0067 |
3.70bpw | 3.70 bpw (3.716), SC allocation | 5 bits | 6 bits | 4 bits | INT8 per row | 13.22 GiB | 0.0089 / 0.0085 |
Unlike our Swift EXL3 quantizations, these were made without YAQA (on Swift, YAQA changed the KLD by 0–3 %).
Files and reproducibility
model-*.safetensors, config.json (quantization_config: exl3, 3.85 bits, head 5, vision 6, MTP 4), quantization_config.json (per-tensor allocation).plugin/qwen38_exl3_vision/: our vLLM plugin (vision tensors and padding). scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.- The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's
qbench_prompts_gen.json, so they can be reproduced
with exllamav3's qbench against the BF16 model.
Results
Family comparison (Qwen3.8-27B, single RTX 3090 24 GB)
Table with columns: Model, Size, Context in 24 GB, KLD mean / median, Perplexity (BF16: 1.5860 · pub. 1.3332), Needle · tools at T=1.0 · T=0.3 · agent, Reasoning xhigh (tokens), Prefill, Decode with MTP, 1 / 2 sessions, Tokens per step, Accepted drafts (of 3)| Model | Size | Context in 24 GB | KLD mean / median | Perplexity (BF16: 1.5860 · pub. 1.3332) | Needle · tools at T=1.0 · T=0.3 · agent | Reasoning xhigh (tokens) | Prefill | Decode with MTP, 1 / 2 sessions | Tokens per step | Accepted drafts (of 3) |
|---|
W4A16 GPTQ + INT4 embeddings (mtp-int8) | 14.2 GiB | 222,000 (text) · 176,128 (vision) |
How to read it:
- KLD: mean / median KL divergence of the next-token distribution against the BF16 Qwen3.8-27B, measured with exllamav3's
qbench. The first
figure uses an internal sysadmin/agent evaluation set (not published; separate from the calibration data, ≈30 % of it with tool calls); pub.
uses turboderp's public qbench_prompts_gen.json trace (from turboderp/Qwen3.8-27B-exl3),
so anyone can reproduce it. Noise floor on the public trace (BF16 against itself): 0.0005. The W4A16 checkpoint was evaluated as a BF16 copy holding
the dequantized INT4 × scale weights (Transformers, full load); the EXL3 ones with exllamav3. Lower is better. Perplexity is on the same texts.
- Context: largest
--max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and ≥100 MiB of VRAM really free at peak.
- Needle: 8 facts spread through one prompt of that length, all 8 returned.
- Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases (fresh request, after a long story, full multi-turn session)
× 20 tries,
xhigh, at T=1.0 (no temperature sent, so vLLM uses the model's generation_config.json) and at T=0.3 (what opencode sends with the
configuration below). The only misses (all at T=1.0) were answers taken from the history ("the folder is empty") instead of a new tool call.
: 4 sysadmin tasks × 2 with and real tool execution.
In short: this EXL3 version is ≈5× closer to the BF16 model than the W4A16 one, at the cost of ≈20 % lower decode speed on Ampere and a more
complex software stack. turboderp's 3.00 bpw is the one that fits the full 262k context with vision, at ≈5× our KLD.
Benchmarks and validated configurations (single RTX 3090 24 GB, 350 W)
Service configuration (quick start above)
Table with columns: Test, main 3.85 bpw (215,000), 3.70bpw (226,000)| Test | main 3.85 bpw (215,000) | 3.70bpw (226,000) |
|---|
| KV pool | 216,433 tokens (KV 8.42e9 B) | 227,439 tokens (KV 8.85e9 B) |
| Idle: used / really free | 23,406 / 719 MiB | 23,346 / 779 MiB |
| 8 facts in one long prompt (greedy) | 8/8 at 200,069 tokens, 371.2 s (539 tok/s) | 8/8 at 210,602 tokens, 400.4 s (526 tok/s) |
| 2 simultaneous sessions | 2 × ≈99k, 8/8 in both, 722 tok/s aggregate | 2 × ≈104k, 8/8 in both, 712 tok/s aggregate |
| Decode, 1 request, t=0 / 0.6 |
Images (EXL3 rows, vision enabled): 4/4 tasks, 18/18 checks with our 2026-10-02 battery (failed-service screenshot, chart, dense firewall ruleset,
and a terminal where the port is held by another process while the journal shows a dead PID as a trap). This battery replaces the older 3-image one
(8/9), whose first image wrongly required that dead PID; the PID is still checked in a follow-up turn.
Reasoning and agent tasks
6 problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, total reasoning tokens over
48 requests; agent = 4 sysadmin tasks × 2 with opencode run (xhigh, temperature 0.3, real tool execution, reasoning counted by a proxy):
Table with columns: Model, low, medium, xhigh, Correct (low · medium · xhigh), Agent: ok · reasoning tokens · time| Model | low | medium | xhigh | Correct (low · medium · xhigh) | Agent: ok · reasoning tokens · time |
|---|
EXL3 3.85 (main) | 18,683 | 20,851 | 23,221 | 47 · 48 · 48 | 8/8 · 5,366 · 342 s |
EXL3 3.70 (3.70bpw) | 17,879 | 20,068 | 21,912 | 48 · 48 · 48 |
The EXL3 versions reason 12–17 % less at xhigh on these problems; on the short agent tasks all reason little and the differences (−24 % to +11 %) are within
the run-to-run spread. The level (low / medium / xhigh) does not regulate the amount of reasoning much in either. For a model that reasons markedly less, see
the Swift 1.5 family.
Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)
Table with columns: Revision, Turn: time to first token / total (median), First turn (no cache), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0 / 0.6), Really free (min)| Revision | Turn: time to first token / total (median) | First turn (no cache) | 8 sessions: needle each, aggregate prefill | Decode with 8 sessions (t=0 / 0.6) | Really free (min) |
|---|
main 3.85 | 3.2 / 4.3 s | 13.7 s | 8 × 23k, 8/8 in all, 972 tok/s | 289.6 / 281.1 tok/s | 149 MiB |
Agent turn = the real opencode request (≈13.5k → 14.7k tokens over 10 turns) with prefix caching. For comparison, the W4A16 sibling needs 2.7 / 3.6 s per turn
and reaches 472 tok/s with 8 sessions, but only up to a 200k pool and without vision.
MTP acceptance
On the mixed workload of the family table: 3.03 tokens per step for main, per-position acceptance 0.86 / 0.66 / 0.52. 3.70bpw has no 8-session run yet;
on the validation run (needle, decode, images, tools) it gave 2.96 tokens per step, against 2.93 for main on the same kind of run.
Sampling and repetition loops
Measured on our EXL3 3.70 build of the same model (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens (a loop = more than 30 % repeated 8-grams).
Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Temperature | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| Loops / 30 | 14 | 9 | 10 | 5 | 4 | 5 | 2 | 0 |
Loops grow as the temperature goes down (14 of 30 at 0.3) and disappear at 1.0: use 1.0, this repo's generation_config.json default, also in opencode. On these long prompts the model often keeps reasoning past 16,000 tokens without answering, so give it a generous output budget. With the real opencode request, tool calls gave the same results at 0.3 and 1.0 (30/30 for Qwen3.8-27B, -pi and Swift 1.5 27B), so the higher temperature costs nothing in agent use. Other models on the same test:
Table with columns: Model (same test, vLLM), T 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Model (same test, vLLM) | T 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| NeoHorse-1-9B | 2 | 3 | 3 | 1 | 2 | 0 | 0 | 0 |
| Ornith-1.5-9B | 5 | 7 |
Requirements & install (vLLM 0.30 + exllamav3 + plugins)
EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:
- Continuum-AI-Corp/OrcaSAQ2-kernel (commit
7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear
layers in vLLM through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed). Its README describes it
as a "correctness path".
qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/; 0.2.0 also adds EXL3 support for MoE experts, unused by this dense model): drops the separate attn.{q,k,v}_proj tensors of the EXL3 vision tower (vLLM uses the fused
attn.qkv) and pads input dimensions to multiples of 128 for layers that need it (4304 → 4352).
Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Qwen3.8-27B-EXL3-3.85bpw-24GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Quick start (single RTX 3090, 24 GB)
main (3.85 bpw), vision + MTP 3, 215k context, 2 concurrent sequences, 1 image per request (≤1 MP) — our validated configuration:
export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Qwen3.8-27B-EXL3-3.85bpw-24GB \ --served-model-name qwen3.8-27b \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 215000 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 8420000000 \ --max-num-batched-tokens 1024 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
3.70bpw branch, 226k context: the same command with --revision 3.70bpw --max-model-len 226000 --kv-cache-memory-bytes 8850000000.
8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈23k tokens at once, or one long session).
Validated on main with 149 MiB really free at peak (not yet run on 3.70bpw).
Using it with opencode (how we tested)
This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was
measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks;
context, speed, memory and image tests were run directly against the vLLM API.
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "qwen3.8-27b": { "name": "Qwen3.8-27B EXL3 (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 215000, "output": 22000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/qwen3.8-27b", "variant": "xhigh", "temperature": 1.0 }, "plan": { "model": "local-vllm/qwen3.8-27b", "variant": "xhigh", "temperature": 1.0 } }}
-
Temperature 1.0 (since 2026-10-13; it was 0.3): our first agent tests of this family used 0.3, but at 0.3 the model falls into repetition loops on long answers (see Sampling and repetition loops), and with the real opencode request tool calls gave the same results at 1.0 as at 0.3.
-
limit.context = --max-model-len (226000 for the 3.70bpw branch); with output: 22000 opencode compacts the history at ≈context − 22k tokens of input.
-
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model (checked by capturing the requests).
The agent tasks ran at temperature 0.3; the tool-call test at both T=1.0 (model default) and T=0.3.
-
Without attachment / modalities opencode will not send images; drop them if you serve with .
Important notes
- Memory. The driver reserves 453 MiB of the 24,576 MiB that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.used,memory.free --format=csv, not as 24,576 − used. Size the KV cache with --kv-cache-memory-bytes, never with
--gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). --max-num-batched-tokens 1024 and
expandable_segments lower the peak by ≈350 MiB at no measurable cost. With MTP, KV capacity grows in block-sized steps: check the GPU KV cache size
line (8,420,000,000 B → 216,433 tokens; 8,850,000,000 B → 227,439) and keep --max-model-len below it.
- Measured peaks (2 sessions, vision): 3.85 bpw 23,814 MiB used, 311 MiB really free; 3.70 bpw 23,754 MiB, 371 MiB free. Leave the GPU for the LLM: a desktop session on the same GPU would eat this margin.
- Hugging Face page warnings: the "Model size" shown by Hugging Face (≈8B params) counts the packed storage elements of the safetensors files
(EXL3 packs ≈4 weights into each
int16), not parameters: this is the 27B model at ≈3.85 bits per weight. The Configuration Parsing Warning
"quantization_config.bits must be an integer" is also expected: for EXL3 is the average bits per weight of a per-tensor allocation
(the per-tensor values are in ); exllamav3 and the OrcaSAQ2 plugin read it as a float.
Quantization recipe
Table with columns: Step, What we did| Step | What we did |
|---|
| Calibration trace | Generated by Qwen3.8-27B itself (reasoning_effort: xhigh, MTP enabled) with exllamav3's sc_trace: 196 prompts: 157 from exllamav3's own sc_trace set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published). The trace is the model's own xhigh answers to them, ≈535k generated tokens; 196 sequences × 2048 tokens used for conversion |
| Reference and sensitivity | sc_rfn_probe against a 2.00 bpw reference, sc_measure --shaped --max-sys 24 (per-tensor sensitivity on the trace) |
| Bit allocation | sc_optimize at the target rate with a 5-bit (, ) → per-tensor recipe (achieved 3.866 / 3.716 bpw) |
- Tokenizer, chat template and generation config unchanged (XML tool calls,
qwen3_coder).
- Hardware: one RTX PRO 6000 Blackwell / L40S per job; conversion peaked at ≈33 GiB of RAM,
sc_measure at ≈57 GiB.
- The measured KLDs were ≈0.55× of
sc_optimize's prediction (3.85: predicted 0.0132, measured 0.0073; 3.70: 0.0162 → 0.0089).
Limitations
- Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images,
short reasoning problems and agent tasks.
- The full 262,144 context does not fit at this rate on 24 GB with MTP and vision (turboderp's 3.00 bpw does, at a higher KLD).
- Decode is ≈20 % slower than our W4A16 GPTQ version on Ampere; the OrcaSAQ2 plugin is young and describes itself as a correctness path.
- The fp8 KV cache uses the default scale 1.0. Vision was validated with synthetic images only (terminal screenshot, chart, dense text), not photos or video.
- All numbers come from one machine and one software stack.
Feedback welcome
I would be grateful for reports of installations of this checkpoint, both working and failed ones, especially on GPUs other than the RTX 3090.
Please open a discussion in the Community tab with your
GPU(s), driver, vLLM / exllamav3 / plugin versions, the full command line and the result (speed, context, or the error message).
Credits & license
Credits & acknowledgements
License
Apache-2.0, same as the base model. This is a community quantization, not an official Qwen release. The OrcaSAQ2 plugin and exllamav3 are not
redistributed here; install them from their repositories under their own licenses.