JRamirez-UAB

Ornith-1.5-9B-EXL3-16GB-12GB

Family comparison (Ornith-1.5-9B, measured on an RTX 3090 within the target memory)

Table with columns: Model, Size, Target GPU · context, KLD mean / median, Perplexity (BF16: 3.1386 · pub. 2.0885), Needle · tools at T=1.0 · T=0.3 · agent, Reasoning xhigh (tokens), Prefill, Decode with MTP, 1 / 2 sessions, Tokens per step, Accepted drafts (of 3)
Model	Size	Target GPU · context	KLD mean / median	Perplexity (BF16: 3.1386 · pub. 2.0885)	Needle · tools at T=1.0 · T=0.3 · agent	Reasoning xhigh (tokens)	Prefill	Decode with MTP, 1 / 2 sessions	Tokens per step	Accepted drafts (of 3)
EXL3 6.00 bpw (`main`, this repo)	7.22 GiB	16 GB · 262,144 (with vision)	0.0019 / 0.0008 · pub. 0.0017 / 0.0003	3.1427 · pub. 2.0896	8/8 at 246,788 · 59/60 · 60/60 · 6/8 (16/16†)	31,415	1,296 tok/s	130.6 / 251.6 tok/s	2.63	1.63
EXL3 4.50 bpw (`4.50bpw`, this repo)	5.79 GiB	12 GB · 198,000 (with vision) / 220,000 (text)	0.0095 / 0.0040 · pub. 0.0086 / 0.0016	3.1447 · pub. 2.0950	8/8 at 184,931 · 58/60 · 60/60 · 6/8 (16/16†)	23,145	1,529 tok/s	137.8 / 255.8 tok/s	2.74	1.74

† The agent misses are always the file-name translation described above; with the file names in backticks, 16/16 (measured on both rates).

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 Ornith-1.5-9B, measured with exllamav3's qbench. The first figure uses an internal sysadmin/agent evaluation set (not published); pub. uses turboderp's public qbench_prompts_gen.json trace, so anyone can reproduce it. Noise floor (BF16 against itself): 0.0009 / 0.0007. Lower is better. Perplexity is on the same texts.
Context: --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and the peak memory used under the target card's budget.
Needle: 8 facts spread through one prompt of that length, all 8 returned. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, xhigh, at T=1.0 (no temperature sent; at the time the repo had no generation_config.json, so vLLM sampled with temperature 1.0 and no top_k / top_p) and at T=0.3 (what opencode sends with the configuration below); the misses at T=1.0 were answers taken from the history instead of a new tool call. Agent: 4 sysadmin tasks × 2 with opencode run and real tool execution.
Reasoning: total reasoning tokens over 6 checkable problems × 8 tries at xhigh (temperature 0.6): 44/48 correct (6.00), 47/48 (4.50).

Variants / revisions

Table with columns: Revision, Target, Linear layers (achieved), lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)
Revision	Target	Linear layers (achieved)	`lm_head`	Vision	MTP head	Embeddings	Size	KLD (ours / pub.)
`main`	16 GB	6.00 bpw (5.999), SC allocation	6 bits	6 bits	6 bits	INT8 per row

For comparison, a uniform 6.00 bpw allocation measured 0.0023 / 0.0021: the per-tensor recipe is ≈18 % closer to BF16 at the same size. We also made 3.50 and 3.00 bpw versions for 8 GB GPUs (KLD 0.032 and 0.065; text only, 46k / 63k tokens of context, peak ≈7,400 MiB used, tools 15/15, reasoning 46–47/48); they are an ongoing project and not published here.

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Calibration trace	Generated by Ornith-1.5-9B itself (`reasoning_effort: xhigh`, MTP enabled) with exllamav3's `sc_trace`: 196 prompts: 157 from exllamav3's own `sc_trace` set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published). The trace is the model's own `xhigh` answers to them, ≈674k generated tokens (277 rows: a second turn in 81 conversations); 250 sequences × 2048 tokens used for conversion
Reference and sensitivity	`sc_rfn_probe` against a 2.00 bpw reference, `sc_measure --shaped --max-sys 16` (per-tensor sensitivity on the trace)
Bit allocation	`sc_optimize` at the target rate ( / , ) → per-tensor recipe

Source: ornith-ai/Ornith-1.5-9B revision 489cb979. Tokenizer, chat template and generation config unchanged (Qwen3.5 XML tool calls, qwen3_coder).
Hardware: one NVIDIA L40S per job; each conversion took ≈20 minutes.

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers in vLLM through exllamav3's kernels, and INT8 embed_tokens for qwen3_5 models (no vLLM patch needed). Its README describes it as a "correctness path".
qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/; 0.2.0 also adds EXL3 support for MoE experts, unused by this dense model): drops the separate attn.{q,k,v}_proj tensors of the EXL3 vision tower (vLLM uses the fused attn.qkv) and pads input dimensions to multiples of 128 for layers that need it. It works for this Qwen3.5-architecture model as well as for Qwen3.8.

Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto        # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision

Quick start

16 GB GPU — main (6.00 bpw), vision + MTP 3, full 262,144 context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB \  --served-model-name ornith-1.5-9b \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 5250000000 \  --max-num-batched-tokens 8192 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

12 GB GPU — 4.50bpw: the same command with --revision 4.50bpw and

with vision: --max-model-len 198000 --kv-cache-memory-bytes 4000000000 --max-num-batched-tokens 2048 (peak 11,500 MiB used; 4096 goes 22 MiB over a 12 GB budget);
text only (drop --limit-mm-per-prompt / --mm-processor-kwargs, add --language-model-only): --max-model-len 220000 --kv-cache-memory-bytes 4430000000 --max-num-batched-tokens 1024 (peak 11,406 MiB used).

Both leave ≈250–300 MiB under the ≈11,700 MiB a 12 GB card offers. A first measurement at 210,000 / 235,000 tokens peaked at 11,684 / 11,706 MiB: it works, but with no margin at all, so we recommend the values above.

8 concurrent sessions: add --max-num-seqs 8 and keep the same KV cache (the pool is shared: 8 × ≈30k tokens at once, or one long session). Validated on 16 GB (peak ≈14,180 MiB used) and on 12 GB with vision at 198,000 (peak ≈11,530 MiB used) — both measured with --max-num-batched-tokens 1024: use 1024 for 8 sessions.

Important notes

File names: use backticks. Ornith-1.5-9B tends to translate file names into English when the prompt is in another language ("escribe errores.txt …" → it writes errors.txt); the content is right. Neither an AGENTS.md instruction nor temperature 0.1 fixed it in our tests (1/4 and 0/4), but writing names, paths and commands in backticks ("escribe `errores.txt` con …") did: 16/16 agent tasks on both rates. This comes from the model, not from the quantization. Measured again on 2026-10-11 with backticks added automatically to every user message: agent success 45/60 → 55/60 (also on ten extra control tasks), 0 renamed files; with the same tasks in English the renaming does not happen at all. It is a non-English problem. The model is not translating: it renames output files whose Spanish name is a system concept (processes, users, errors) to the conventional English name; names unrelated to the system (e.g. tortuga.txt) are kept (34/34). Backticks fix it and did not hurt when the user describes a system file in words or misspells its path (70/70).
Memory. The driver reserves ≈450 MiB that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). expandable_segments lowers the peak at no measurable cost. --max-num-batched-tokens trades peak memory for prompt speed: 8192 on 16 GB reads a 124k-token prompt ≈27 % faster than 1024 (2,235 vs 1,762 tok/s on the RTX 3090) with a peak of 15,256 MiB used (validated: needle 8/8 at 233k, 2 sessions, vision, tools); on 12 GB use 2048 (+6 %). With MTP, KV capacity grows in block-sized steps: check the line (5,250,000,000 B → 267,185 tokens) and keep below it.

Using it with opencode (how we tested)

This checkpoint is meant to be used as a coding / sysadmin agent from opencode (tested with opencode 1.18.33). Tool calling was measured with the real opencode request (captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run against the vLLM API.

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "ornith-1.5-9b": {          "name": "Ornith 1.5 9B EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 32000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.6 },    "plan":  { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.6 }  }}

limit.context = --max-model-len (use the 12 GB value with the 4.50bpw branch); opencode compacts the history at ≈context − output tokens of input.
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model. Drop attachment / modalities for text-only serving.
Start opencode inside a project directory; for opencode run from scripts, close stdin (</dev/null).

Benchmarks and validated configurations (RTX 3090, 350 W, within the target memory budget)

Table with columns: Test, main 6.00 bpw, 16 GB, 262,144 + vision, 4.50bpw, 12 GB, 198,000 + vision, 4.50bpw, 12 GB, 220,000 text only
Test	`main` 6.00 bpw, 16 GB, 262,144 + vision	`4.50bpw`, 12 GB, 198,000 + vision	`4.50bpw`, 12 GB, 220,000 text only
KV pool	267,185 tokens (KV 5.25e9 B)	199,980 tokens (KV 4.0e9 B)	222,986 tokens (KV 4.43e9 B)
8 facts in one long prompt (greedy)	8/8 at 246,788 tokens, 190 s (1,296 tok/s)	8/8 at 184,931 tokens, 121 s (1,529 tok/s)	8/8 at 206,162 tokens, 143 s (1,445 tok/s)
2 simultaneous sessions	2 × ≈125k, 8/8 in both

Images: failed-service screenshot, chart, dense firewall ruleset, and a terminal where a port is held by another process while the journal shows a dead PID as a trap. The missed check, when there is one, is always the same: after freeing the port the model restarts the service without checking the port again first.

Reasoning and agent tasks

6 problems with checkable answers (arithmetic, word problem, counting, logic, subnetting, systemd) × 8 tries, temperature 0.6, total reasoning tokens over 48 requests; agent = 4 sysadmin tasks × 2 with opencode run (xhigh, temperature 0.3, real tool execution):

Table with columns: Revision, low, medium, xhigh, Correct (low · medium · xhigh), Agent (plain prompts), Agent (file names in backticks)
Revision	low	medium	xhigh	Correct (low · medium · xhigh)	Agent (plain prompts)	Agent (file names in backticks)
`main` 6.00	32,636	23,479	31,415	46 · 43 · 44	6/8 (both misses: `errors.txt`)	16/16 (earlier 6.00 build)
`4.50bpw`	22,839

For reference, on the same 48 problems Qwen3.5-9B used ≈146k reasoning tokens at xhigh (39/48, 9 answers cut off) and a 27B Qwen3.8 ≈18–23k.

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)

Table with columns: Revision, Turn: time to first token / total (median), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0), Peak used
Revision	Turn: time to first token / total (median)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0)	Peak used
`main` 6.00 (16 GB)	0.76 / 1.4 s	8 × 29k, 8/8 in all, 2,824 tok/s	556 tok/s	≈14,180 MiB
`4.50bpw` (12 GB, 198,000 + vision)	0.75 / 1.3 s	8 × 22k, 8/8 in 7 and 7/8 in one, 2,931 tok/s	591 tok/s	≈11,530 MiB

Limitations

Measured on an RTX 3090 within the memory of the target cards, not yet on actual 16 GB / 12 GB GPUs (their bandwidth and compute differ: expect lower speeds).
Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images, short reasoning problems and agent tasks. Ornith's own benchmarks are on the BF16 model.
The file-name translation quirk above. The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.
Invented command options and fields (measured 2026-10-11): with less common CLIs (Slurm sacctmgr/scontrol, nvidia-smi --query-*) it may guess field names or subcommands that do not exist; check its commands. An instruction to read --help first did not fix it. The Slurm and systemd-timer tasks are the ones it misses most often. This is a knowledge limit of a 9B model, the same in Spanish and English; documenting the tools your agent uses (field names, correct syntax) in its instructions is the practical workaround.
Reasoning length: short. On our 6 problems × 3 it used ≈8,400–8,700 reasoning tokens per run (median ≈500 per answer) with 16–18/18 correct; reasoning_effort low/medium/xhigh made no difference in how much it thinks.
is its weak spot (see above): prefer a larger model for coding.

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3), quantization_config.json (per-tensor allocation); SHA256SUMS.
plugin/qwen38_exl3_vision/: our vLLM plugin. scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's qbench_prompts_gen.json, so they can be reproduced with exllamav3's qbench against the BF16 model.
NOTICE: license and attribution of the original model.

Feedback welcome

I would be grateful for reports of installations of this checkpoint, especially on real 16 GB and 12 GB GPUs, both working and failed ones. Please open a discussion in the Community tab with your GPU, driver, vLLM / exllamav3 / plugin versions, the full command line and the result (speed, context, peak memory, or the error message).

Credits & acknowledgements

Base model: ornith-ai/Ornith-1.5-9B by the Ornith team (built on the Qwen3.5 family).
exllamav3 and the EXL3 format by turboderp (MIT), including the SC pipeline, qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.

License

MIT, as declared by the original model card (the original repository does not include a LICENSE file; see NOTICE). This is a community quantization, not an official release of the Ornith team. The OrcaSAQ2 plugin and exllamav3 are not redistributed here; install them from their repositories under their own licenses.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

ornith-ai/Ornith-1.5-9B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Family comparison (Ornith-1.5-9B, measured on an RTX 3090 within the target memory)

Table with columns: Model, Size, Target GPU · context, KLD mean / median, Perplexity (BF16: 3.1386 · pub. 2.0885), Needle · tools at T=1.0 · T=0.3 · agent, Reasoning xhigh (tokens), Prefill, Decode with MTP, 1 / 2 sessions, Tokens per step, Accepted drafts (of 3)
Model	Size	Target GPU · context	KLD mean / median	Perplexity (BF16: 3.1386 · pub. 2.0885)	Needle · tools at T=1.0 · T=0.3 · agent	Reasoning xhigh (tokens)	Prefill	Decode with MTP, 1 / 2 sessions	Tokens per step	Accepted drafts (of 3)
EXL3 6.00 bpw (`main`, this repo)	7.22 GiB	16 GB · 262,144 (with vision)	0.0019 / 0.0008 · pub. 0.0017 / 0.0003	3.1427 · pub. 2.0896	8/8 at 246,788 · 59/60 · 60/60 · 6/8 (16/16†)	31,415	1,296 tok/s	130.6 / 251.6 tok/s	2.63	1.63
EXL3 4.50 bpw (`4.50bpw`, this repo)	5.79 GiB	12 GB · 198,000 (with vision) / 220,000 (text)	0.0095 / 0.0040 · pub. 0.0086 / 0.0016	3.1447 · pub. 2.0950	8/8 at 184,931 · 58/60 · 60/60 · 6/8 (16/16†)	23,145	1,529 tok/s	137.8 / 255.8 tok/s	2.74	1.74

† The agent misses are always the file-name translation described above; with the file names in backticks, 16/16 (measured on both rates).

How to read it:

KLD: mean / median KL divergence of the next-token distribution against the BF16 Ornith-1.5-9B, measured with exllamav3's qbench. The first figure uses an internal sysadmin/agent evaluation set (not published); pub. uses turboderp's public qbench_prompts_gen.json trace, so anyone can reproduce it. Noise floor (BF16 against itself): 0.0009 / 0.0007. Lower is better. Perplexity is on the same texts.
Context: --max-model-len validated with MTP 3, 2 sequences, fp8 KV cache and the peak memory used under the target card's budget.
Needle: 8 facts spread through one prompt of that length, all 8 returned. Tools: the real request of the opencode agent (system prompt + 11 tools), 3 cases × 20 tries, xhigh, at T=1.0 (no temperature sent; at the time the repo had no generation_config.json, so vLLM sampled with temperature 1.0 and no top_k / top_p) and at T=0.3 (what opencode sends with the configuration below); the misses at T=1.0 were answers taken from the history instead of a new tool call. Agent: 4 sysadmin tasks × 2 with opencode run and real tool execution.
Reasoning: total reasoning tokens over 6 checkable problems × 8 tries at xhigh (temperature 0.6): 44/48 correct (6.00), 47/48 (4.50).

Variants / revisions

Table with columns: Revision, Target, Linear layers (achieved), lm_head, Vision, MTP head, Embeddings, Size, KLD (ours / pub.)
Revision	Target	Linear layers (achieved)	`lm_head`	Vision	MTP head	Embeddings	Size	KLD (ours / pub.)
`main`	16 GB	6.00 bpw (5.999), SC allocation	6 bits	6 bits	6 bits	INT8 per row

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Calibration trace	Generated by Ornith-1.5-9B itself (`reasoning_effort: xhigh`, MTP enabled) with exllamav3's `sc_trace`: 196 prompts: 157 from exllamav3's own `sc_trace` set (logic, maths, code, science; English and Spanish) and 39 of our own generic sysadmin/agent conversations (invented examples, no real system data; not published). The trace is the model's own `xhigh` answers to them, ≈674k generated tokens (277 rows: a second turn in 81 conversations); 250 sequences × 2048 tokens used for conversion
Reference and sensitivity	`sc_rfn_probe` against a 2.00 bpw reference, `sc_measure --shaped --max-sys 16` (per-tensor sensitivity on the trace)
Bit allocation	`sc_optimize` at the target rate ( / , ) → per-tensor recipe

Source: ornith-ai/Ornith-1.5-9B revision 489cb979. Tokenizer, chat template and generation config unchanged (Qwen3.5 XML tool calls, qwen3_coder).
Hardware: one NVIDIA L40S per job; each conversion took ≈20 minutes.

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

EXL3 is not a native vLLM format. Two plugins are needed, both loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc): EXL3 linear layers in vLLM through exllamav3's kernels, and INT8 embed_tokens for qwen3_5 models (no vLLM patch needed). Its README describes it as a "correctness path".
qwen38_exl3_vision (ours, version 0.2.0, in plugin/qwen38_exl3_vision/; 0.2.0 also adds EXL3 support for MoE experts, unused by this dense model): drops the separate attn.{q,k,v}_proj tensors of the EXL3 vision tower (vLLM uses the fused attn.qkv) and pads input dimensions to multiples of 128 for layers that need it. It works for this Qwen3.5-architecture model as well as for Qwen3.8.

Python 3.12 is required (the exllamav3 wheels are cp312), plus a CUDA 13.2 toolkit (nvcc) for flashinfer's JIT kernels on first start:

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.30.0 --torch-backend=auto        # torch 2.13.0+cu132W=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc)hf download JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB --include "plugin/*" --local-dir .uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_vision

Quick start

16 GB GPU — main (6.00 bpw), vision + MTP 3, full 262,144 context, 2 concurrent sequences, 1 image per request (≤1 MP):

bash

export CUDA_HOME=/usr/local/cuda-13.2 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Ornith-1.5-9B-EXL3-16GB-12GB \  --served-model-name ornith-1.5-9b \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 5250000000 \  --max-num-batched-tokens 8192 \  --max-num-seqs 2 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":3}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

12 GB GPU — 4.50bpw: the same command with --revision 4.50bpw and

with vision: --max-model-len 198000 --kv-cache-memory-bytes 4000000000 --max-num-batched-tokens 2048 (peak 11,500 MiB used; 4096 goes 22 MiB over a 12 GB budget);
text only (drop --limit-mm-per-prompt / --mm-processor-kwargs, add --language-model-only): --max-model-len 220000 --kv-cache-memory-bytes 4430000000 --max-num-batched-tokens 1024 (peak 11,406 MiB used).

Important notes

File names: use backticks. Ornith-1.5-9B tends to translate file names into English when the prompt is in another language ("escribe errores.txt …" → it writes errors.txt); the content is right. Neither an AGENTS.md instruction nor temperature 0.1 fixed it in our tests (1/4 and 0/4), but writing names, paths and commands in backticks ("escribe `errores.txt` con …") did: 16/16 agent tasks on both rates. This comes from the model, not from the quantization. Measured again on 2026-10-11 with backticks added automatically to every user message: agent success 45/60 → 55/60 (also on ten extra control tasks), 0 renamed files; with the same tasks in English the renaming does not happen at all. It is a non-English problem. The model is not translating: it renames output files whose Spanish name is a system concept (processes, users, errors) to the conventional English name; names unrelated to the system (e.g. tortuga.txt) are kept (34/34). Backticks fix it and did not hurt when the user describes a system file in words or misspells its path (70/70).
Memory. The driver reserves ≈450 MiB that vLLM cannot use: read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). expandable_segments lowers the peak at no measurable cost. --max-num-batched-tokens trades peak memory for prompt speed: 8192 on 16 GB reads a 124k-token prompt ≈27 % faster than 1024 (2,235 vs 1,762 tok/s on the RTX 3090) with a peak of 15,256 MiB used (validated: needle 8/8 at 233k, 2 sessions, vision, tools); on 12 GB use 2048 (+6 %). With MTP, KV capacity grows in block-sized steps: check the line (5,250,000,000 B → 267,185 tokens) and keep below it.

Using it with opencode (how we tested)

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "ornith-1.5-9b": {          "name": "Ornith 1.5 9B EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 32000 },          "variants": {            "none":   { "chat_template_kwargs": { "enable_thinking": false } },            "low":    { "chat_template_kwargs": { "reasoning_effort": "low" } },            "medium": { "chat_template_kwargs": { "reasoning_effort": "medium" } },            "xhigh":  { "chat_template_kwargs": { "reasoning_effort": "xhigh" } }          }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.6 },    "plan":  { "model": "local-vllm/ornith-1.5-9b", "variant": "xhigh", "temperature": 0.6 }  }}

limit.context = --max-model-len (use the 12 GB value with the 4.50bpw branch); opencode compacts the history at ≈context − output tokens of input.
Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model. Drop attachment / modalities for text-only serving.
Start opencode inside a project directory; for opencode run from scripts, close stdin (</dev/null).

Benchmarks and validated configurations (RTX 3090, 350 W, within the target memory budget)

Table with columns: Test, main 6.00 bpw, 16 GB, 262,144 + vision, 4.50bpw, 12 GB, 198,000 + vision, 4.50bpw, 12 GB, 220,000 text only
Test	`main` 6.00 bpw, 16 GB, 262,144 + vision	`4.50bpw`, 12 GB, 198,000 + vision	`4.50bpw`, 12 GB, 220,000 text only
KV pool	267,185 tokens (KV 5.25e9 B)	199,980 tokens (KV 4.0e9 B)	222,986 tokens (KV 4.43e9 B)
8 facts in one long prompt (greedy)	8/8 at 246,788 tokens, 190 s (1,296 tok/s)	8/8 at 184,931 tokens, 121 s (1,529 tok/s)	8/8 at 206,162 tokens, 143 s (1,445 tok/s)
2 simultaneous sessions	2 × ≈125k, 8/8 in both

Reasoning and agent tasks

Table with columns: Revision, low, medium, xhigh, Correct (low · medium · xhigh), Agent (plain prompts), Agent (file names in backticks)
Revision	low	medium	xhigh	Correct (low · medium · xhigh)	Agent (plain prompts)	Agent (file names in backticks)
`main` 6.00	32,636	23,479	31,415	46 · 43 · 44	6/8 (both misses: `errors.txt`)	16/16 (earlier 6.00 build)
`4.50bpw`	22,839

For reference, on the same 48 problems Qwen3.5-9B used ≈146k reasoning tokens at xhigh (39/48, 9 answers cut off) and a 27B Qwen3.8 ≈18–23k.

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)

Table with columns: Revision, Turn: time to first token / total (median), 8 sessions: needle each, aggregate prefill, Decode with 8 sessions (t=0), Peak used
Revision	Turn: time to first token / total (median)	8 sessions: needle each, aggregate prefill	Decode with 8 sessions (t=0)	Peak used
`main` 6.00 (16 GB)	0.76 / 1.4 s	8 × 29k, 8/8 in all, 2,824 tok/s	556 tok/s	≈14,180 MiB
`4.50bpw` (12 GB, 198,000 + vision)	0.75 / 1.3 s	8 × 22k, 8/8 in 7 and 7/8 in one, 2,931 tok/s	591 tok/s	≈11,530 MiB

Limitations

Measured on an RTX 3090 within the memory of the target cards, not yet on actual 16 GB / 12 GB GPUs (their bandwidth and compute differ: expect lower speeds).
Quality was not evaluated with standard benchmarks; the checks are KL divergence against BF16, long-context retrieval, tool calling, vision on synthetic images, short reasoning problems and agent tasks. Ornith's own benchmarks are on the BF16 model.
The file-name translation quirk above. The fp8 KV cache uses the default scale 1.0. All numbers come from one machine and one software stack.
Invented command options and fields (measured 2026-10-11): with less common CLIs (Slurm sacctmgr/scontrol, nvidia-smi --query-*) it may guess field names or subcommands that do not exist; check its commands. An instruction to read --help first did not fix it. The Slurm and systemd-timer tasks are the ones it misses most often. This is a knowledge limit of a 9B model, the same in Spanish and English; documenting the tools your agent uses (field names, correct syntax) in its instructions is the practical workaround.
Reasoning length: short. On our 6 problems × 3 it used ≈8,400–8,700 reasoning tokens per run (median ≈500 per answer) with 16–18/18 correct; reasoning_effort low/medium/xhigh made no difference in how much it thinks.
is its weak spot (see above): prefer a larger model for coding.

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3), quantization_config.json (per-tensor allocation); SHA256SUMS.
plugin/qwen38_exl3_vision/: our vLLM plugin. scripts/embed_int8.py: re-encodes embed_tokens to INT8 per row.
The calibration and evaluation sets are internal and not published; the public KLD figures use turboderp's qbench_prompts_gen.json, so they can be reproduced with exllamav3's qbench against the BF16 model.
NOTICE: license and attribution of the original model.

Feedback welcome

Credits & acknowledgements

Base model: ornith-ai/Ornith-1.5-9B by the Ornith team (built on the Qwen3.5 family).
exllamav3 and the EXL3 format by turboderp (MIT), including the SC pipeline, qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.

Ornith-1.5-9B-EXL3-16GB-12GB

README

Family comparison (Ornith-1.5-9B, measured on an RTX 3090 within the target memory)

Variants / revisions

Quantization recipe

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

Quick start

Important notes

Using it with opencode (how we tested)

Benchmarks and validated configurations (RTX 3090, 350 W, within the target memory budget)

Reasoning and agent tasks

Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)

Limitations

Files and reproducibility

Feedback welcome

Credits & acknowledgements

License

Explore FriendliAI today

README

Family comparison (Ornith-1.5-9B, measured on an RTX 3090 within the target memory)

Variants / revisions

Quantization recipe

Requirements & install (vLLM 0.30 + exllamav3 + plugins)

Quick start

Important notes

Using it with opencode (how we tested)

Benchmarks and validated configurations (RTX 3090, 350 W, within the target memory budget)

Reasoning and agent tasks

Agent turns and 8 sessions (--max-num-seqs 8, same KV pool)

Limitations

Files and reproducibility

Feedback welcome

Credits & acknowledgements

License

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)

Agent turns and 8 sessions (`--max-num-seqs 8`, same KV pool)