What is in this repository
Which one to use
Table with columns: GPU, Configuration, Context per session, Real concurrent sessions, KV cache, Speculative decoding, Pick it for| GPU | Configuration | Context per session | Real concurrent sessions | KV cache | Speculative decoding | Pick it for |
|---|
| 24 GB (RTX 3090) | this repo, several users | 262,144 (text) | 4 × 236k measured, --max-num-seqs 16 | int4, 1.54 M tokens | none | a shared server: several agents on one card |
| 24 GB (RTX 3090) | this repo, vision | 262,144 with vision | 3 × 262k by vLLM's count (not load-tested) | k8v4_per_token_head, 825,631 tokens | MTP 2 | images, one or two users |
| 24 GB (RTX 3090) | branch 3.50bpw (16.83 GB), several users | 262,144 (text) | 4 × 236k measured | int4, 1.20 M tokens | none | lower KLD (0.0096 / 0.0108) and better programming; worse agent and tool-call results here |
| 16 GB | — | — | — | — | — | does not fit: 14.82 GB of weights |
For our own sysadmin agent we serve Ornith-1.5-35B-A3B EXL3 3.00, a sibling quantization with the same recipe and the same 24 GB budget: it scored higher in our agent, programming and tool tests (see Known model behaviour).
Files and reproducibility
Table with columns: File, What it is| File | What it is |
|---|
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json | EXL3 weights (14.82 GB) and their index |
quantization_config.json | exllamav3's per-tensor storage record (bits per tensor, INT4 embedding layout; calibration size only, no text) |
config.json, generation_config.json | model config; sampling defaults from the base model card (temperature 1.0, top_k 20, top_p 0.95, presence_penalty 1.5) |
chat_template.jinja, tokenizer.json, tokenizer_config.json | chat template and tokenizer of the base model |
preprocessor_config.json, processor_config.json, video_preprocessor_config.json | image/video processor of the base model |
LICENSE, NOTICE | Apache-2.0 text and the list of changes made to the base model |
SHA256SUMS | SHA-256 of every file above except this card |
Not included: the BF16 reference, the calibration data (internal), the plugin (install it from PyPI, see below).
Results
Unless stated otherwise: one RTX 3090 24 GB (no other use), vLLM 0.31.0 + exllamav3 1.6.5 + vllm-quantpack 0.1.4, --max-model-len 262144, --kv-cache-dtype int4_per_token_head, --max-num-seqs 8 for the battery, measured on 2026-11-02 with the several users settings.
Quantization quality (exllamav3 qbench, against BF16)
Table with columns: Trace, KLD mean, KLD median, KLD p90, Perplexity (BF16), EXL3 3.50, same recipe: KLD mean · perplexity| Trace | KLD mean | KLD median | KLD p90 | Perplexity (BF16) | EXL3 3.50, same recipe: KLD mean · perplexity |
|---|
| opencode agent trace (ours, 24 conversations) | 0.0152 | 0.0053 | 0.0394 | 3.2160 (3.1954) | 0.0096 · 3.2118 |
public (qbench_prompts_gen.json, turboderp) | 0.0171 | 0.0025 | 0.0481 | 1.9941 (1.9784) | 0.0108 · 1.9857 |
Measured on these exact files (INT4 embedding included, exllamav3 1.6.7, 2026-11-17); with BF16 embeddings the same conversion measured 0.0147 / 0.0167. For reference, our Ornith-1.5-35B-A3B 3.00 with the same recipe measures 0.0332 / 0.0366 against its own BF16 on the same traces.
Capacity on one RTX 3090 (24 GB)
Table with columns: Configuration, --kv-cache-memory-bytes, KV pool reported by vLLM, Real concurrent sessions, Peak used| Configuration | --kv-cache-memory-bytes | KV pool reported by vLLM | Real concurrent sessions | Peak used |
|---|
| several users: text, no MTP, int4 KV, 8192-token chunks | 9018005258 | 1,543,326 tokens | 4 sessions of 235,929 tokens at once, 0 preemptions | 23,742 MiB, 374 MiB free |
vision: MTP 2, k8v4_per_token_head KV, 4 sequences | 8100000000 | 825,631 tokens (3.15 × 262,144) | not load-tested | 23,472 MiB |
With 9,405,978,378 bytes the card ran out of memory, so the several users value is the largest that kept ≥300 MiB free.
Test battery (several users configuration)
Table with columns: Test, Result| Test | Result |
|---|
Tool calls, real opencode request, 3 cases × 20 (xhigh) | 51/60 correct (20/20, 15/20, 16/20), 53 valid calls, 0 errors |
| Tool calls, battery | 28/30 valid and correct, 0 errors |
Agent tasks, opencode run, 10 sysadmin tasks × 3 runs | 8/10, 8/10, 8/10 = 24/30 |
| Programming tasks (12 × 3) | 11/12, 11/12, 10/12 (1,460-1,700 s per run) |
| Needle, 9 neutral facts | 64k 9/9 · 128k 9/9 · 255k 9/9 |
| Needle, 8 facts in one prompt | 64k 8/8 · 128k 8/8 · 241k 8/8 |
| Decode, 1 / 2 / 4 / 8 requests | 94.1 / 106.2 / 242.5 / 344.7 tok/s |
| Prefill / decode near 15k · 64k · 128k · 260k | 5,322 / 91.6 · 4,111 / 80.7 · 3,055 / 69.6 · 2,011 / 54.8 tok/s |
| Agents at once (tasks passed, median time per task) | 4: 8/8, 78 s · 8: 16/16, 148 s · 16: 27/32, 301 s (6 timeouts) |
| Time to first token of a new 32k prompt while 16 agents run | median 7.7 s |
Vision (vision configuration, vLLM 0.34.0 + vllm-quantpack 0.1.6, 2026-11-17)
Table with columns: Test, Result| Test | Result |
|---|
| Our 4-image test (screenshots of a failing systemd unit, a chart, an nftables listing and a port conflict) | 15/15 |
| Second turn about the same image (prefix cache) | correct, 0.7 s |
| One image after a 151,013-token prompt | correct, 58.9 s |
| Peak VRAM used | 23,506 MiB |
Speed with and without MTP (vLLM 0.34.0 + vllm-quantpack, 2026-11-17)
One request at a time, 512 generated tokens per prompt, the card's sampling (temperature 1.0, top_p 0.95, top_k 20, presence_penalty 1.5), k8v4_per_token_head KV with an 8,100,000,000-byte pool, text only. Acceptance = accepted / proposed draft tokens.
Table with columns: Prompt, No MTP: decode, MTP 1: decode (acceptance), MTP 2: decode (acceptance)| Prompt | No MTP: decode | MTP 1: decode (acceptance) | MTP 2: decode (acceptance) |
|---|
| 14,847 tokens | 88.6 tok/s | 104.5 tok/s (0.63) | 122.9 tok/s (0.54) |
| 64,211 tokens | 77.6 tok/s | 95.0 tok/s (0.69) | 110.0 tok/s (0.57) |
| 128,753 tokens | 66.3 tok/s | 79.6 tok/s (0.65) | 85.9 tok/s (0.48) |
| 260,954 tokens | 51.8 tok/s | 60.4 tok/s (0.63) | 71.8 tok/s (0.54) |
| KV pool | 963,764 tokens | 847,514 tokens | 823,527 tokens |
| Lowest free VRAM | 1,748 MiB | 1,146 MiB | 1,002 MiB |
Prefill near 64k / 128k / 261k: 3,928 / 3,029 / 1,995 tok/s without MTP and 3,798 / 2,803 / 1,827 tok/s with MTP 2. With 4 sessions of 15k at once, total decode was about 188 tok/s without MTP, 129 with MTP 1 and 143 with MTP 2: MTP pays off for one or two users, not for a shared server.
Sampling and repetition loops
5 long Spanish sysadmin questions × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20, several users settings with 8 sequences; a loop = more than 30 % repeated word 8-grams.
Table with columns: Temperature, 0.6, 0.8, 1.0| Temperature | 0.6 | 0.8 | 1.0 |
|---|
With presence_penalty 1.5 (vLLM 0.34.0, 2026-11-17): loops / 30 | 3 | 0 | 0 |
| Answers that finished before 16,000 tokens / 30 | 27 | 28 | 30 |
| Empty answers / 30 | 3 | 2 | 0 |
Without it (vLLM 0.34.0, presence_penalty 0 sent explicitly, 2026-11-17): loops / 30 | 9 | 8 | 2 |
| Without it: answers that finished / 30 | 21 | 22 | 28 |
| Without it: empty answers / 30 | 9 | 8 | 2 |
| Without it, earlier stack (vLLM 0.31.0, 2026-11-02): loops / 30 | 10 | 5 | 1 |
The control without the penalty ran on the same engine version, so the drop in loops comes from presence_penalty 1.5 and not from the newer vLLM. A loop here usually ends as an empty answer: the model keeps repeating inside its reasoning until the 16,000-token limit. Note that a request that omits presence_penalty gets 1.5 from this repository's generation_config.json; to measure without it, send 0.
Use temperature 1.0 with presence_penalty 1.5, as the base model card recommends. generation_config.json sets both and vLLM uses them as the server defaults; a client that sends its own presence_penalty (or temperature) overrides them.
Known model behaviour and discarded options
- Against our Ornith 3.00 (same recipe, same card, same tests): Sage has half the KL divergence against its own BF16, but in our sysadmin tests it scored lower: agent tasks 24/30 against 27/30, programming 32/36 against 36/36 (and about twice as slow per run), tool calls with the real opencode request 51/60 against 59/60, 16 agents at once 27/32 against 28/32. The needle (9/9 at every depth) and vision (15/15) were the same.
- Weak case in tool calls: with a long user prompt followed by a request to list files, 15 of 20 calls were correct (Ornith: 20/20).
- Loops without
presence_penalty: see the table above.
- Discarded options:
- EXL3 3.50 (same recipe, 16.83 GB, branch
3.50bpw, load it with --revision 3.50bpw): lower KLD (0.0096 / 0.0108) and 35/36 in programming, but a smaller pool (1.20 M tokens), 7/10 × 3 in agent tasks, and only 5/20 correct in the weak tool-call case. Not the default here; its own card on the branch has the full results.
- MTP for several users: with 4 sessions at once MTP 2 gave less total decode than no MTP (about 143 against 188 tok/s) and it shrinks the KV pool, so the several users configuration runs without it.
Requirements & install (vLLM 0.34.0 + exllamav3 1.6.7 + vllm-quantpack 0.1.6)
Before you install (checked on our Ubuntu 26.04 host on 2026-11-17):
- An NVIDIA driver that supports CUDA 13.2 (tested with 595.91), uv and git.
- The CUDA toolkit 13.2 (
nvcc): exllamav3 1.6.7 publishes no wheel for torch 2.14 (the torch of vLLM 0.34), so it is built from source below (about 20 minutes with 8 parallel jobs), and FlashInfer 0.7.3 has no prebuilt kernels yet, so it compiles them the first time vLLM starts.
- A C compiler (
build-essential). We use uv's own Python (--managed-python), which includes the headers Triton needs.
- Disk: about 20 GB for the venv, as much again for uv's cache (
uv cache clean frees it), plus 15 GB for the model.
EXL3 is not a native vLLM format. vllm-quantpack (ours, on PyPI) loads it through vLLM's plugin entry point: EXL3 linear, embedding (INT4/INT8) and MoE layers through exllamav3's kernels, the vision tower, the MTP head, a fixed int4 KV cache and the k8v4_per_token_head KV type. Version 0.1.6 or later is required (k8v4_per_token_head, and the vision tower on vLLM 0.34).
uv venv --python 3.12 --managed-python venvuv pip install --python venv/bin/python vllm==0.34.0 --torch-backend=auto # torch 2.14.0+cu132uv pip install --python venv/bin/python ninja packaging setuptools wheelgit clone --depth 1 --branch v1.6.7 https://github.com/turboderp-org/exllamav3 exllamav3-srcexport CUDA_HOME=/usr/local/cuda-13.2 PATH=/usr/local/cuda-13.2/bin:$PATHTORCH_CUDA_ARCH_LIST="8.6" MAX_JOBS=8 uv pip install --python venv/bin/python --no-build-isolation ./exllamav3-src # 8.6 = RTX 3090; add yoursuv pip install --python venv/bin/python "vllm-quantpack>=0.1.6"venv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Quick start (single RTX 3090, 24 GB)
Several users (the configuration of the battery, capacity and loop tables; text only, no MTP):
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Sage-A1.5-35B-A3B-EXL3-3.00bpw-24GB \ --served-model-name sage \ --language-model-only \ --max-model-len 262144 \ --kv-cache-dtype int4_per_token_head \ --kv-cache-memory-bytes 9018005258 \ --max-num-seqs 16 \ --max-num-batched-tokens 8192 \ --enable-prefix-caching \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
Vision with MTP 2:
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Sage-A1.5-35B-A3B-EXL3-3.00bpw-24GB \ --served-model-name sage \ --max-model-len 262144 \ --kv-cache-dtype k8v4_per_token_head \ --kv-cache-memory-bytes 8100000000 \ --max-num-seqs 4 \ --max-num-batched-tokens 8192 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
On a card that also drives a display, lower --kv-cache-memory-bytes by what the desktop uses. On 48 GB GPUs raise it: the KV of this model is ≈5.8 KB per token in int4 (only 10 of the 40 layers are full attention, with 2 KV heads).
Using it with opencode (how we tested)
limit.context must match --max-model-len (262144 above).
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "sage": { "name": "Sage-A1.5 EXL3 3.00 (local)", "tool_call": true, "temperature": true, "limit": { "context": 262144, "output": 32768 } } } } }, "agent": { "build": { "model": "local-vllm/sage", "temperature": 1.0 }, "plan": { "model": "local-vllm/sage", "temperature": 1.0 } }}
- Declare
"temperature": true in the model, otherwise opencode sends no temperature to a custom model.
- opencode sends no
presence_penalty, so the server default from generation_config.json (1.5) applies.
- With
output: 32768 opencode compacts the history at about context − 32k tokens of input, before vLLM rejects the request.
- The agent and tool tests in Results ran opencode against the several users server with this configuration, without
presence_penalty (they were measured before the loop test): the agent results with it are not measured.
Important notes
VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 is required. The fused exllamav3 MoE kernel needs the whole GPU; otherwise vLLM runs the shared expert on a parallel CUDA stream and decoding hangs. The plugin sets it, but export it yourself too.
- Memory. The driver reserves about 453 MiB of a 24 GB card that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.free. Size the KV cache with --kv-cache-memory-bytes, not --gpu-memory-utilization.
- Prefix caching matters for agents: opencode sends the same tools and system prompt (about 9k tokens) in every request, and each turn repeats the whole conversation, so keep each conversation on the same server when you run several.
- Long prompts: the first turn of a 260k-token prompt takes about 2 minutes 10 seconds to prefill on a 3090; later turns reuse the cache.
- Reasoning counts towards
max_tokens: with a small max_tokens the answer can come back empty; allow ≥ 2,000 tokens or disable reasoning with chat_template_kwargs: {"enable_thinking": false}.
- Hugging Face page warnings: the "Model size" shown by Hugging Face counts packed storage elements, not parameters; the Configuration Parsing Warning about
quantization_config.bits not being an integer is expected (EXL3 bits is an average).
- No tensor or expert parallelism for the EXL3 MoE path: one GPU per instance.
- Use
--tool-call-parser qwen3_coder and --reasoning-parser qwen3.
Quantization recipe
Table with columns: Step, Setting| Step | Setting |
|---|
| Source | InternScience/Sage-A1.5 revision 0e3205cf; chat template, tokenizer and processors unchanged |
| Tool | exllamav3 1.6.5 convert.py, RTX PRO 6000 Blackwell |
| Routed experts | 3.00 bpw (-b 3.00) |
| Attention, linear attention, shared experts | 4 bits (-hq 1) |
lm_head / vision tower / MTP head | 6 / 6 / 4 bits |
| Calibration | 589 rows built from about 600 agent steps written by the BF16 model on our internal sysadmin prompts plus real conversation prefixes; not published, and no text is stored in the weights |
| Token embeddings | INT4 per row with BF16 scales: python -m vllm_quantpack.tools.embed_quant SRC DST --bits 4 on the converted checkpoint |
Integrity and validation
Artifact integrity
SHA256SUMS lists every file; the verifier compares the repo with our local copy byte for byte before publication.
- The safetensors files carry no
__metadata__; quantization_config.json holds only bit widths, tensor layout and the calibration size (rows and columns).
Quantization
- KL divergence against the BF16 model with exllamav3's
qbench on two traces, on these exact files (Results).
Runtime
- Load, 262k needle, capacity, tool calls with the real opencode request, agent and programming tasks, vision and loops: see Results.
- This repository downloaded from Hugging Face and served with the several users Quick start (vLLM 0.34.0, 2026-11-17): KV pool 1,557,884 tokens, 1,579 MiB free at the lowest point, no engine errors; 4 long Spanish answers with the default sampling and at temperature 0: no repetition and no CJK characters, 3 of the 4 reached our 3,000-token check limit (the model writes long answers).
Scope: results apply to the stacks, the RTX 3090 and the settings stated above; other setups are not tested.
Limitations
- Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine where you can.
- Will not load with stock vLLM, Transformers or SGLang: it needs exllamav3 and vllm-quantpack.
- Not measured: the agent and tool tests with
presence_penalty 1.5, the request-field check with opencode captures, concurrency in the vision configuration, reasoning problems at xhigh, a detailed vision table (only our 4-image test), GPUs other than the RTX 3090 for serving, and the base model's scientific benchmarks on this quantization. The calibration and the opencode trace are internal, so those KLD numbers cannot be reproduced outside; the public-trace KLD can.
- The EXL3 MoE path, the int4 KV fixes and
k8v4_per_token_head are our own plugin code, not upstream vLLM.
Feedback welcome
Reports of installations, both working and failed ones, are very welcome in the Community tab: GPU(s), driver, vLLM version, the full command line and the result (speed, context, or the error).
Credits & license
- Base model: InternScience/Sage-A1.5 by Shanghai AI Laboratory (Apache-2.0), built on Qwen/Qwen3.6-35B-A3B.
- exllamav3 by turboderp (quantization, kernels,
qbench and the public trace); EXL3 is based on QTIP (Tseng et al.).
- vLLM; vllm-quantpack (ours, Apache-2.0).
- This quantization is released under the same license as the base model (Apache-2.0); the changes are listed in
NOTICE.