What is in this repository
Which one to use
All Ornith-1.5-9B variants we publish, measured on 2026-10-14 under the same conditions (see Family comparison for every number):
Table with columns: Variant, Weights, Context on one RTX 3090, KLD vs BF16, Agent tasks, Decode, 1 request, Pick it for| Variant | Weights | Context on one RTX 3090 | KLD vs BF16 | Agent tasks | Decode, 1 request | Pick it for |
|---|
| EXL3 6.00 (16 GB) (this repo) | 7.20 GiB | 262,144 with vision | 0.0019 | 23/30 | 136.5 tok/s | a 16 GB GPU: 262,144 tokens with vision |
| EXL3 4.50 (12 GB) (this repo) | 5.77 GiB | 198,000 with vision | 0.0095 | 20/30 | 138.0 tok/s | a 12 GB GPU: 198,000 tokens with vision |
Variants / revisions
Table with columns: Revision, Target, Linear layers (achieved), lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)| Revision | Target | Linear layers (achieved) | lm_head | Vision | MTP head | Embeddings | Size | KLD (ours / pub.) |
|---|
main | 16 GB | 6.00 bpw (5.999), SC allocation | 6 bits | 6 bits | 6 bits | INT8 per row | 7.20 GiB | 0.0019 / 0.0017 |
4.50bpw | 12 GB | 4.50 bpw (4.500), SC allocation | 5 bits | 4 bits | 6 bits | INT8 per row | 5.77 GiB | 0.0095 / 0.0086 |
For comparison, a uniform 6.00 bpw allocation measured 0.0023 / 0.0021: the per-tensor recipe is ≈18 % closer to BF16 at the same size.
We also made 3.50 and 3.00 bpw versions for 8 GB GPUs (KLD 0.032 and 0.065; text only, 46k / 63k tokens of context, peak ≈7,400 MiB used, tools 15/15,
reasoning 46–47/48); they are an ongoing project and not published here.
Files and reproducibility
model-*.safetensors, config.json (quantization_config: exl3), quantization_config.json (per-tensor allocation); SHA256SUMS.
plugin/qwen38_exl3_vision/: our vLLM plugin. scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
- The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's
qbench_prompts_gen.json, so they can be reproduced
with exllamav3's qbench against the BF16 model.
NOTICE: license and attribution of the original model.
Results
Family comparison (Ornith-1.5-9B, same conditions)
Measured on 2026-10-14 for every variant in the same way: weights downloaded from Hugging Face, one RTX 3090 24 GB (power limit 350 W, no other use), vLLM 0.31.0 (EXL3: + exllamav3 1.5.1 + plugins), each variant with its own Quick start configuration. KLD and perplexity come from exllamav3's qbench (see How to read it).
Table with columns: Variant, Weights, KLD mean / median (internal · public), Perplexity (BF16 3.1395 · pub. 2.0882), Needle (8 facts) near max. context, Tools: real opencode request, Agent tasks (10 × 3), Reasoning xhigh: correct · tokens, Vision| Variant | Weights | KLD mean / median (internal · public) | Perplexity (BF16 3.1395 · pub. 2.0882) | Needle (8 facts) near max. context | Tools: real opencode request | Agent tasks (10 × 3) | Reasoning xhigh: correct · tokens | Vision |
|---|
EXL3 6.00 bpw (main, this repo) | 7.20 GiB | 0.0019 / 0.0008 · pub. 0.0017 / 0.0003 | 3.1427 · pub. 2.0896 | 8/8 at 241,437 | 60/60 | 23/30 | 45/48 · 27,067 | 14/15 |
EXL3 4.50 bpw (4.50bpw, this repo) | 5.77 GiB | 0.0095 / 0.0040 · pub. 0.0086 / 0.0016 | 3.1447 · pub. 2.0950 | 8/8 at 182,413 | 60/60 | 20/30 | 45/48 · 29,708 | 14/15 |
Table with columns: Variant, Context on one RTX 3090, Sessions, Prefill near max. context, Decode, 1 / 2 / 8 requests (tok/s), MTP tokens per step, VRAM used at peak (fits a 16 GB card up to ≈15,700 MiB, a 12 GB card up to ≈11,700)| Variant | Context on one RTX 3090 | Sessions | Prefill near max. context | Decode, 1 / 2 / 8 requests (tok/s) | MTP tokens per step | VRAM used at peak (fits a 16 GB card up to ≈15,700 MiB, a 12 GB card up to ≈11,700) |
|---|
| EXL3 6.00 (16 GB) | 262,144 with vision | 2 | 1,694 tok/s | 136.5 / 240.6 / 252.3 | 2.88 | 15,232 MiB |
| EXL3 4.50 (12 GB) | 198,000 with vision | 2 | 1,665 tok/s | 138.0 / 249.7 / 265.0 | 2.84 | 11,560 MiB |
How to read it:
- KLD / perplexity: next-token KL divergence against the BF16 model with exllamav3's
qbench (lower is better). Internal: our sysadmin/agent evaluation set (not published, separate from the calibration data); public: turboderp's qbench_prompts_gen.json, reproducible by anyone. GPTQ checkpoints were evaluated as a BF16 copy holding the dequantized weights. These two columns come from qbench, not from this campaign.
- Needle: 8 facts spread through one prompt at about 92 % of the served context; all 8 must come back. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, at the model's default sampling (
generation_config.json); a try counts when it calls a valid tool with valid arguments and the expected command. Agent: 10 sysadmin tasks run with opencode run and real command execution, 3 runs, at the recommended temperature; with sampling on, the same checkpoint varies by about ±2 of 30 between runs, so other measurements in this card may differ by that much. Reasoning: 6 checkable problems × 8 at xhigh; total reasoning tokens over the 48. Vision: our 4-image test.
- Prefill / decode: prefill from the needle prompt; decode is aggregate throughput with 1 / 2 / 8 simultaneous requests (temperature 0; with more requests than Sessions, the extra ones wait). MTP tokens per step: vLLM's speculative-decoding metrics over the whole run.
- VRAM really free: lowest
memory.free while serving (nvidia-smi every 0.5 s, from the end of start-up to the end of the run); we require ≥100 MiB.
- Older sections below keep their own date and setup; use this table to compare variants.
How many sessions fit with each Quick start configuration (KV size in tokens as reported by vLLM at start-up; a session uses the KV of its own context, and vLLM never runs more than max sessions at once):
Table with columns: Variant, KV pool (tokens), Max sessions, 32k each, 64k each, 128k each, Full context each| Variant | KV pool (tokens) | Max sessions | 32k each | 64k each | 128k each | Full context each |
|---|
| EXL3 6.00 (16 GB) | 267,185 | 2 | 2 | 2 | 2 | 1 × 262,144 |
| EXL3 4.50 (12 GB) | 199,980 | 2 | 2 | 2 | 1 | 1 × 198,000 |
Few users or many users. The Quick start configuration is tuned for 1–2 simultaneous users (fastest single request). For more users, the many users option below (measured the same day, same GPU) gives more total throughput; each request gets slower when many run at once. Same model, same quality: only the serving settings change.
Table with columns: Variant, Configuration, Decode by simultaneous requests (aggregate tok/s), N sessions at once, needle 8/8, Tools (5 × 3), VRAM used at peak| Variant | Configuration | Decode by simultaneous requests (aggregate tok/s) | N sessions at once, needle 8/8 | Tools (5 × 3) | VRAM used at peak |
|---|
| EXL3 6.00 (16 GB) | Quick start (2 sessions, MTP 3) | 1: 136 · 2: 241 · 8: 252 | — | 60/60 | 15,232 MiB |
| EXL3 6.00 (16 GB) | many users (8 sessions of up to 262,144 tokens, MTP 3) | 1: 131 · 2: 240 · 4: 474 · 8: 579 | 6/8 of 22k | 15/15 | 15,442 MiB |
| EXL3 4.50 (12 GB) | Quick start (2 sessions, MTP 3) | 1: 138 · 2: 250 · 8: 265 | — | 60/60 | 11,560 MiB |
| EXL3 4.50 (12 GB) | many users (8 sessions of up to 198,000 tokens, MTP 3) | 1: 144 · 2: 253 · 4: 497 · 8: 644 | 5/8 of 22k | 15/15 | 11,688 MiB |
Benchmarks and validated configurations (RTX 3090, 350 W, within the target memory budget)
Earlier measurements, before the controlled comparison of 2026-10-14: other dates, vLLM versions or settings, as stated in each table. Use Family comparison above to compare variants.
Table with columns: Test, main 6.00 bpw, 16 GB, 262,144 + vision, 4.50bpw, 12 GB, 198,000 + vision, 4.50bpw, 12 GB, 220,000 text only| Test | main 6.00 bpw, 16 GB, 262,144 + vision | 4.50bpw, 12 GB, 198,000 + vision | 4.50bpw, 12 GB, 220,000 text only |
|---|
| KV pool | 267,185 tokens (KV 5.25e9 B) | 199,980 tokens (KV 4.0e9 B) | 222,986 tokens (KV 4.43e9 B) |
| 8 facts in one long prompt (greedy) | 8/8 at 246,788 tokens, 190 s (1,296 tok/s) | 8/8 at 184,931 tokens, 121 s (1,529 tok/s) | 8/8 at 206,162 tokens, 143 s (1,445 tok/s) |
| 2 simultaneous sessions | 2 × ≈125k, 8/8 in both | 2 × ≈91k, 8/8 in both | 2 × ≈102k, 7/8 in both |
| Decode, 1 request / 2 concurrent (t=0) | 130.6 / 251.6 tok/s | 137.8 / 255.8 tok/s | 127.4 / 251.8 tok/s |
| Images (4 synthetic tasks, 18 checks; two runs) | 18/18 and 17/18 | 17/18 and 17/18 | — |
| Real opencode request, tool calls (3 cases × 20; T=1.0 / T=0.3) | 59/60 / 60/60 | 58/60 / 60/60 | 60/60 / 60/60 |
| Peak memory used | 14,072 MiB (16 GB budget ≈15,700) | 11,444 MiB (12 GB budget ≈11,700) | 11,406 MiB |
Images: failed-service screenshot, chart, dense firewall ruleset, and a terminal where a port is held by another process while the journal shows a dead PID as a trap.
The missed check, when there is one, is always the same: after freeing the port the model restarts the service without checking the port again first.
Reasoning and agent tasks
6 problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, total reasoning tokens over 48 requests;
agent = 4 sysadmin tasks × 2 with opencode run (xhigh, temperature 0.3, our setting until 2026-10-13, real tool execution):
Table with columns: Revision, low, medium, xhigh, Correct (low · medium · xhigh), Agent (plain prompts), Agent (file names in backticks)| Revision | low | medium | xhigh | Correct (low · medium · xhigh) | Agent (plain prompts) | Agent (file names in backticks) |
|---|
main 6.00 | 32,636 | 23,479 | 31,415 | 46 · 43 · 44 | 6/8 (both misses: errors.txt) | 16/16 (earlier 6.00 build) |
4.50bpw | 22,839 | 27,007 | 23,145 | 45 · 45 · 47 | 6/8 (both misses: errors.txt) | 16/16 |
For reference, on the same 48 problems Qwen3.5-9B used ≈146k reasoning tokens at xhigh (39/48, 9 answers cut off) and a 27B Qwen3.8 ≈18–23k.
Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)
Table with columns: Revision, Turn: time to first token / total (median), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0), Peak used| Revision | Turn: time to first token / total (median) | 8 sessions: needle each, aggregate prefill | Decode with 8 sessions (t=0) | Peak used |
|---|
main 6.00 (16 GB) | 0.76 / 1.4 s | 8 × 29k, 8/8 in all, 2,824 tok/s | 556 tok/s | ≈14,180 MiB |
4.50bpw (12 GB, 198,000 + vision) | 0.75 / 1.3 s | 8 × 22k, 8/8 in 7 and 7/8 in one, 2,931 tok/s | 591 tok/s | ≈11,530 MiB |
Sampling and repetition loops
Measured on 2026-10-14 with the same software and GPU as Family comparison, on our EXL3 6.00 (16 GB) build of Ornith-1.5-9B (loops come from the model and the sampling, not from the quantization format): 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated 8-grams. Serving: the Quick start with a 32k context and 8 sessions.
Table with columns: Temperature, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Temperature | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| Loops / 30 | 6 | 2 | 3 | 1 | 1 | 0 | 0 | 0 |
| Answers that finished before 16,000 tokens / 30 | 25 | 23 | 26 | 25 | 25 | 22 | 24 | 20 |
From temperature 0.6 up, loops are rare (at most 1 of 30). This repo's default (generation_config.json) is 0.8.
Other models, same test and setup (loops / 30):
Table with columns: Model (build), 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0| Model (build) | 0.3 | 0.4 | 0.5 | 0.6 | 0.7 | 0.8 | 0.9 | 1.0 |
|---|
| Qwen3.8-27B · EXL3 3.85 | 12 | 12 | 7 | 8 | 1 | 1 | 1 | 0 |
| Qwen3.8-27B-pi · EXL3 3.85 | 9 | 4 | 9 | 10 | 4 | 2 | 1 | 0 |
| Qwopus3.8-27B-Flash-V2 · EXL3 3.85 | 3 | 1 | 2 | 1 | 2 | 0 | 0 | 0 |
| Swift 1.5 (Qwen3.8-27B) · EXL3 3.85 | 7 | 5 | 3 | 4 | 3 | 1 | 1 | 0 |
| NeoHorse-1-9B · EXL3 6.00 (16 GB) | 6 | 3 | 3 | 1 | 0 | 0 | 0 | 0 |
| Qwen3.6-35B-A3B · EXL3 4.00 | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 |
| Ornith-1.5-35B-A3B · EXL3 4.00 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Qwopus3.6-35B-A3B-Coder · EXL3 4.00 | 9 | 13 | 12 | 6 | 11 | 6 | 5 | 4 |
| Swift 1.5 Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM | 24 | 24 | 21 | 21 | 17 | 8 | 0 | 0 |
| Qwen3.8-Flash-Next (125B-A6B) · EXL3 2.25 · vLLM | — | — | 13 | 7 | 5 | 2 | 0 | 0 |
Requirements & install (vLLM 0.31 + exllamav3 1.5.1 + plugins)
EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:
- Continuum-AI-Corp/OrcaSAQ2-kernel (commit
7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear
layers through exllamav3's kernels, and INT8 / packed embed_tokens and lm_head for qwen3_5 models (no vLLM patch needed).
qwen38_exl3_vision 0.4.1 (ours; 0.4.1 is a maintenance release; 0.4.0 added multi-GPU: tensor parallelism for the dense models, -tp 2, e.g. Swift 27B on 2 GPUs with the same log-likelihood (+0.04 %) and +43 % speed for one request, and expert parallelism for the MoE ones, -tp 2 --enable-expert-parallel; 0.3.2 freed the host-RAM copy of the checkpoint, ~14 GB less for a 27B; in plugin/qwen38_exl3_vision/, the same version in all our EXL3 repos): the EXL3 vision
tower (drops the separate attn.{q,k,v}_proj tensors and pads input dimensions to multiples of 128 where needed), EXL3 MoE experts with exllamav3's
fused kernel, and the Qwen3.8-Flash-Next model.
Use exllamav3 1.5.1 (with 1.6.x the KLD is the same, but tool calls without reasoning got worse in our tests) and Python 3.12 (the exllamav3 wheels
are cp312). flashinfer compiles some kernels on first start with the CUDA toolkit that comes with the vLLM wheel; if your system nvcc does not match,
point CUDA_HOME at a CUDA 13.0 toolkit.
uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto # torch 2.13.0W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"
Quick start
16 GB GPU — main (6.00 bpw), vision + MTP 3, full 262,144 context, 2 concurrent sequences, 1 image per request (≤1 MP):
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB \ --served-model-name ornith-1.5-9b \ --limit-mm-per-prompt '{"image":1,"video":0}' \ --mm-processor-kwargs '{"max_pixels":1003520}' \ --max-model-len 262144 \ --kv-cache-dtype fp8_e4m3 \ --kv-cache-memory-bytes 5250000000 \ --max-num-batched-tokens 8192 \ --max-num-seqs 2 \ --enable-prefix-caching \ --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \ --reasoning-parser qwen3 \ --enable-auto-tool-choice --tool-call-parser qwen3_coder
12 GB GPU — 4.50bpw: the same command with --revision 4.50bpw and
- with vision:
--max-model-len 198000 --kv-cache-memory-bytes 4000000000 --max-num-batched-tokens 2048 (peak 11,500 MiB used; 4096 goes 22 MiB over a 12 GB budget);
- text only (drop
--limit-mm-per-prompt / --mm-processor-kwargs, add --language-model-only): --max-model-len 220000 --kv-cache-memory-bytes 4430000000
--max-num-batched-tokens 1024 (peak 11,406 MiB used).
Both leave ≈250–300 MiB under the ≈11,700 MiB a 12 GB card offers. A first measurement at 210,000 / 235,000 tokens peaked at 11,684 / 11,706 MiB: it works,
but with no margin at all, so we recommend the values above.
8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈30k tokens at once, or one long session).
Validated on 16 GB (peak ≈14,180 MiB used) and on 12 GB with vision at 198,000 (peak ≈11,530 MiB used) — both measured with --max-num-batched-tokens 1024: use 1024 for 8 sessions.
For many users (measured on 2026-10-14, same GPU): the same command with --max-num-seqs 8 — 8 sessions of up to 262,144 tokens, MTP 3. Aggregate decode 579 tok/s with 8 requests (Quick start: 252 with 8); a single request runs at 131 tok/s (136 with the Quick start); 6 of 8 simultaneous sessions returned all 8 needle facts; 15,442 MiB of VRAM used at peak. Rule for CUDA graphs: they must cover sessions × (1 + MTP) tokens per step.
Using it with opencode (how we tested)
This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was measured with
the real opencode request (captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run against the vLLM API.
{ "$schema": "https://opencode.ai/config.json", "provider": { "local-vllm": { "npm": "@ai-sdk/openai-compatible", "name": "local vLLM", "options": { "baseURL": "http://127.0.0.1:8000/v1" }, "models": { "ornith-1.5-9b": { "name": "Ornith 1.5 9B EXL3 (local)", "tool_call": true, "temperature": true, "attachment": true, "modalities": { "input": ["text", "image"], "output": ["text"] }, "limit": { "context": 262144, "output": 32000 }, "variants": { "none": { "chat_template_kwargs": { "enable_thinking": false } }, "low": { "chat_template_kwargs": { "reasoning_effort": "low" } }, "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } }, "xhigh": { "chat_template_kwargs": { "reasoning_effort": "xhigh" } } } } } } }, "agent": { "build": { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.8 }, "plan": { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.8 } }}
limit.context = --max-model-len (use the 12 GB value with the 4.50bpw branch); opencode compacts the history at ≈context − output tokens of input.
- Declare
"temperature": true in the model, otherwise opencode sends no temperature to a custom model. Drop attachment / modalities for text-only serving.
- Start opencode inside a project directory; for
opencode run from scripts, close stdin (</dev/null).
Important notes
- File names: use backticks. Ornith-1.5-9B tends to translate file names into English when the prompt is in another language ("escribe errores.txt …" →
it writes
errors.txt); the content is right. Neither an AGENTS.md instruction nor temperature 0.1 fixed it in our tests (1/4 and 0/4), but writing names,
paths and commands in backticks ("escribe `errores.txt` con …") did: 16/16 agent tasks on both rates. This comes from the model, not from the quantization.
Measured again on 2026-10-11 with backticks added automatically to every user message: agent success 45/60 → 55/60 (also on ten extra control tasks),
0 renamed files; with the same tasks in English the renaming does not happen at all. It is a non-English problem.
The model is not translating: it renames output files whose Spanish name is a system concept (processes, users, errors) to the conventional English name; names unrelated to the system (e.g. tortuga.txt) are kept (34/34). Backticks fix it and did not hurt when the user describes a system file in words or misspells its path (70/70).
- Memory. The driver reserves ≈450 MiB that vLLM cannot use: read the real headroom with
nvidia-smi --query-gpu=memory.used,memory.free --format=csv. Size the
KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the
activations). expandable_segments lowers the peak at no measurable cost. --max-num-batched-tokens trades peak memory for prompt speed: 8192 on 16 GB reads a 124k-token prompt
≈27 % faster than 1024 (2,235 vs 1,762 tok/s on the RTX 3090) with a peak of 15,256 MiB used (validated: needle 8/8 at 233k, 2 sessions, vision, tools); on 12 GB use 2048 (+6 %). With MTP, KV capacity grows in block-sized steps:
check the GPU KV cache size line (5,250,000,000 B → 267,185 tokens) and keep --max-model-len below it.
- Leave the GPU to the LLM. On a 12 GB card the margin is a few hundred MiB: a desktop session or a browser on the same GPU will not fit.
- Hugging Face page warnings: the "Model size" shown by Hugging Face counts the packed storage elements of the safetensors files (EXL3 packs several weights into
each
int16), not parameters: this is the 9B model. The Configuration Parsing Warning "quantization_config.bits must be an integer" is also expected: for EXL3
bits is the average bits per weight of a per-tensor allocation.
VLLM_USE_FLASHINFER_SAMPLER=0 is recommended by the OrcaSAQ2 plugin. Use --tool-call-parser qwen3_coder and --reasoning-parser qwen3. reasoning_effort accepts
xhigh, medium and low; they are prompt instructions, not limits.
Quantization recipe
Table with columns: Step, What we did| Step | What we did |
|---|
| Calibration trace | Generated by Ornith-1.5-9B itself (reasoning_effort: xhigh, MTP enabled) with exllamav3's sc_trace: 196 prompts: 157 from exllamav3's own sc_trace set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published). The trace is the model's own xhigh answers to them, ≈674k generated tokens (277 rows: a second turn in 81 conversations); 250 sequences × 2048 tokens used for conversion |
| Reference and sensitivity | sc_rfn_probe against a 2.00 bpw reference, sc_measure --shaped --max-sys 16 (per-tensor sensitivity on the trace) |
| Bit allocation | sc_optimize at the target rate (-hb 6 / -hb 5, -al 2.0) → per-tensor recipe |
| Conversion | exllamav3 1.5.1 convert.py with the recipe and the calibration trace; vision 6 / 4 bits (-vb), MTP 6 bits (-mb 6); codebook mul1, output scales always |
| Embeddings | embed_tokens re-encoded to INT8 per row (absmax / 127, BF16 scales; 0.8 % relative error) in the OrcaSAQ2 plugin's format (≈1 GiB less VRAM); script included |
- Source:
ornith-ai/Ornith-1.5-9B revision 489cb979. Tokenizer, chat template and generation config unchanged (Qwen3.5 XML tool calls, qwen3_coder).
- Hardware: one NVIDIA L40S per job; each conversion took ≈20 minutes.
Integrity and validation
What was checked for this checkpoint, how, and where the numbers are (measured on 2026-10-14; the method and every figure are in Family comparison).
Artifact integrity
- Every weight file is checked against our local copy with SHA-256 before publication (
SHA256SUMS in this repo).
- The measurements used the weights downloaded from this repo at commit
68757fe18e8e; later commits change only metadata, documentation or the plugin, never the .safetensors files.
- vLLM loads every tensor listed in
model.safetensors.index.json on start-up (a missing tensor or a wrong shape would stop it): checked from a fresh download.
- After the metadata update of 2026-10-14 (
quantization_config.json, plugin 0.3.1, embed_int8.py): downloaded again from this repo and re-tested: loads; vision 14/15 needle 64k 8/8; tool calls 15/15; engine errors 0.
Quantization
- Per-tensor bit allocation is recorded by exllamav3 in
quantization_config.json; its tensor_storage entry for embed_tokens describes the INT8 qweight + BF16 scales actually stored (updated on 2026-10-14; the plugin only reads the .trellis entries).
- The INT8 embedding conversion (
scripts/embed_int8.py) is reproducible: re-running it on the source checkpoint gives the published qweight and scales bit for bit (checked on Ornith-1.5-9B 6.00 bpw); its reported error now uses the stored BF16 scales (0.00806 vs 0.00803 with FP32 scales).
- Fidelity: KL divergence against the BF16 Ornith-1.5-9B with exllamav3's
qbench on two traces (ours and turboderp's public one).
Chat template
- The published template renders correctly in 40 edge cases (empty and very long system prompt, no tools, two tool calls in one message, nested JSON and multi-line arguments, images with tools, two images, earlier reasoning, multi-turn with several tool results) × the reasoning settings (
enable_thinking: false, reasoning_effort low / xhigh): all pass (prueba_plantillas.py). With Transformers directly, pass tool-call arguments as dicts (vLLM converts JSON strings itself).
Runtime
- Load, long-context needle, vision, tool calls with the real opencode request, 10 agent tasks × 3 runs, concurrency and VRAM: see Family comparison; engine errors during the run: 0.
Scope: results apply to the revision, software stack (vLLM 0.31.0 + exllamav3 1.5.1 + plugins), hardware and settings stated above; other setups are not tested.
Limitations
- Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: review them before running anything as root or with
sudo, and give the agent a non-privileged account and a test machine where you can.
- Measured on an RTX 3090 within the memory of the target cards, not yet on actual 16 GB / 12 GB GPUs (their bandwidth and compute differ: expect lower speeds).
- Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images,
short reasoning problems and agent tasks. Ornith's own benchmarks are on the BF16 model.
- The file-name translation quirk above. The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.
- Invented command options and fields (measured 2026-10-11): with less common CLIs (Slurm
sacctmgr/scontrol, nvidia-smi --query-*) it may guess field
names or subcommands that do not exist; check its commands. An instruction to read --help first did not fix it. The Slurm and systemd-timer tasks are the ones
it misses most often. This is a knowledge limit of a 9B model, the same in Spanish and English; documenting the tools your agent uses (field names,
correct syntax) in its instructions is the practical workaround.
- Reasoning length: short. On our 6 problems × 3 it used ≈8,400–8,700 reasoning tokens per run (median ≈500 per answer) with 16–18/18 correct;
reasoning_effort low/medium/xhigh made no difference in how much it thinks.
- Code that runs is its weak spot (see above): prefer a larger model for coding.
Feedback welcome
I would be grateful for reports of installations of this checkpoint, especially on real 16 GB and 12 GB GPUs, both working and failed ones. Please open a discussion in
the Community tab with your GPU, driver, vLLM / exllamav3 / plugin versions, the full
command line and the result (speed, context, peak memory, or the error message).
Credits & license
Credits & acknowledgements
- Base model: ornith-ai/Ornith-1.5-9B by the Ornith team (built on the Qwen3.5 family).
- exllamav3 and the EXL3 format by turboderp (MIT), including the SC pipeline,
qbench and the public evaluation trace;
EXL3 is based on QTIP (Tseng et al.).
- vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
License
MIT, as declared by the original model card (the original repository does not include a LICENSE file; see NOTICE). This is a community quantization, not an
official release of the Ornith team. The OrcaSAQ2 plugin and exllamav3 are not redistributed here; install them from their repositories under their own licenses.