JRamirez-UAB

Groot-19B-A3B-EXL3-4.00bpw-16-24GB

README

License: apache-2.0

What is in this repository

Which one to use

Table with columns: GPU size, Context per session, Full-context sessions, KV cache, Speculative decoding
GPU size	Context per session	Full-context sessions	KV cache	Speculative decoding
16 GB	212,992	1 (pool of 214,928 tokens)	fp8, 2,655,707,463 B	MTP 1
24 GB	262,144	3 (pool of 927,285 tokens)	fp8, 11,296,602,849 B	MTP 1

On 16 GB the full 262,144 tokens do not fit (the first attempt with a 3.83 GB KV cache went 759 MiB over).

Variants / revisions

Table with columns: Revision, Routed experts, Attention + shared expert, lm_head, Vision, MTP, Embeddings, Size, KLD (sysadmin / opencode / public)
Revision	Routed experts	Attention + shared expert	`lm_head`	Vision	MTP	Embeddings	Size	KLD (sysadmin / opencode / public)
`main`	4.00 bpw	6 bits	6 bits	6 bits	4 bits	BF16	11.5 GB	0.0108 / 0.0100 / 0.0105

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3), quantization_config.json (per-tensor storage), tokenizer files, chat_template.jinja, the processor configs and generation_config.json (unchanged: temperature 1.0, top_p 0.95, top_k 20, the authors' recommended sampling).
plugin/qwen38_exl3_vision/: our vLLM plugin (EXL3 vision tower and MTP head for the Qwen3.5 / Qwen3.6 architectures).
patches/orcasaq2-7ddffb12.diff: the changes our tested OrcaSAQ2 has on top of commit 7ddffb12. setup.sh: the install below as one script.
LICENSE (Apache-2.0, from the base model), SHA256SUMS.
Not included: the BF16 original, exllamav3, vLLM and OrcaSAQ2 (installed from their repositories), our calibration and evaluation traces.

Results

Quantization quality

KL divergence and perplexity against the BF16 original with exllamav3's qbench (2026-10-26). Traces: our own Spanish sysadmin trace, a trace built from real opencode sessions, and turboderp's public qbench_prompts_gen.json.

Table with columns: Trace, KLD mean, KLD median, KLD p90, PPL BF16, PPL EXL3
Trace	KLD mean	KLD median	KLD p90	PPL BF16	PPL EXL3
Our sysadmin trace	0.0108	0.0038	0.0276	3.5345	3.5407
opencode sessions	0.0100	0.0035	0.0251	4.3234	4.3219
turboderp's public trace	0.0105	0.0017	0.0290	2.3879	2.3912

How the conversion was chosen (all at 4.00 bpw on the experts, same KLD traces):

Table with columns: Conversion, Size, KLD sysadmin / opencode / public, Decode, 1 / 8 requests (no MTP, RTX PRO 6000)
Conversion	Size	KLD sysadmin / opencode / public	Decode, 1 / 8 requests (no MTP, RTX PRO 6000)
plain `-b 4.00`	11.1 GB	0.0212 / 0.0196 / 0.0203	116 / 618 tok/s
`-hq 1` (this repo)	11.5 GB	0.0108 / 0.0100 / 0.0105	108 / 590 tok/s
`-hq 2` (attention + shared expert at 8 bits)	11.8 GB	0.0101 / 0.0093 / 0.0100	101 / 583 tok/s
`-hq 1` + calibration on our sysadmin conversations	11.5 GB	0.0103 / 0.0094 / 0.0108	—

The attention and the shared expert are a small part of the weights but run for every token: at 6 bits they halve the KLD for 3 % more size and 7 % less single-request decode. 8 bits or domain calibration add little.

Capacity per GPU size

Vision (1 image, ≤1 MP) and MTP 1 enabled, KV fp8, --max-num-batched-tokens 1024. "Pool" is the GPU KV cache size vLLM reports at start-up; "full-context sessions" = pool ÷ context, rounded down; "concurrent" = the peak of running requests with sessions of 90 % of the context sent at once. On the L40S the limit of each size assumes the 460 MiB a real GPU of that size reserves for the driver.

Table with columns: GPU, Size, --max-model-len, --kv-cache-memory-bytes, KV pool (tokens), Full-context sessions, Concurrent, VRAM peak, Really free at peak
GPU	Size	`--max-model-len`	`--kv-cache-memory-bytes`	KV pool (tokens)	Full-context sessions	Concurrent	VRAM peak	Really free at peak
L40S	16 GB	212,992	2,655,707,463	214,928	1	1	15,543 MiB	381 MiB
L40S	24 GB	262,144	11,296,602,849	927,285	3	4	23,713 MiB	403 MiB
RTX 3090 (real)	24 GB	262,144	11,337,035,939	931,198	3	4	23,450 MiB	~670 MiB

The RTX 3090 row ran vLLM 0.32.0rc1 (nightly) + exllamav3 1.6.1 with --attention-backend FLASHINFER: in that vLLM the Triton attention backend refuses an fp8 KV cache below SM89. The 3090 is also the desktop's display GPU; the start-up used a more conservative target, so its free margin is larger.

Test battery

262,144-token context, KV fp8 with the 24 GB size, MTP 1, --max-num-seqs 8 (the first run used 4: tools, reasoning, vision, needle and games come from it), temperature 1.0 (top_p 0.95, top_k 20), L40S, 2026-10-26. The BF16 original was run on an L40S with the same settings and a 65,536-token context for the tests where the quantization was in doubt; its needle numbers come from an earlier run with thinking off.

Table with columns: Test, EXL3 4.00 -hq, BF16 original
Test	EXL3 4.00 `-hq`	BF16 original
Smoke (short answer; 4 long Spanish answers, 8,000-token limit)	all OK (3,222–4,627 tokens); no CJK (in the two end-to-end runs below, 2 CJK or full-width characters in 1 of 8 long answers)	—
Speed (decode, 1 / 2 / 4 / 8 requests)	139 / 249 / 435 / 716 tok/s	—
Tool calls, real opencode request	battery 17/30; replay 34/60 (20/20 full session, 2/20 "prompt + ls", 12/20 "greeting + story + ls")	battery 16/30; replay 32/60 (20 / 3 / 9)
Reasoning (48 problems)	40/48, 50,399 reasoning tokens (median 1,063), none cut	—
Vision (4 cases, 15 checks)	13/15	—
Needle (8 facts)	64k 7/8 · 128k 6/8 · 241k 5/8	64k 5/8 · 128k 5/8 · 241k 5/8
Needle with thinking, ~250k (3 runs: T 0, T 1.0, T 1.0 seed 7)	7/8 · 7/8 · 8/8 (the T 0 miss is one digit wrong: GEZ-5650 for GEZ-5660)	8/8 · 7/8 · 8/8
Concurrent sessions (8 × 30k)	0 of 8 returned all 8 facts (5–7 each)	0 of 8
Agent tasks (opencode, 10 sysadmin tasks × 3 runs)	7/10 · 7/10 · 7/10 = 21/30 (276–347 s per run)	—
Games	39/48 points, 1 of 4 run	41/48, 1 of 4 run (an earlier run: 43/48, 3 of 4)

Short-context recall without thinking is the model's weak spot, not the quantization's: the BF16 original also misses facts in the 30k sessions and in the battery needle (which sends enable_thinking: false), and finds them when it reasons; so does this EXL3 (22 of 24 facts at ~250k with thinking, BF16 23 of 24). In tool calling it often answers in text instead of calling the tool when the request is a bare "ls" after opencode's long system prompt, BF16 included.

Speculative decoding

RTX 3090 (24 GB, real), 262,144-token context, --max-num-seqs 8, KV fp8, vLLM 0.32.0rc1 + exllamav3 1.6.1 + FlashInfer.

Table with columns: Variant, 1 / 2 / 4 / 8 requests (tok/s), Acceptance
Variant	1 / 2 / 4 / 8 requests (tok/s)	Acceptance
none	96.6 / 156.8 / 263.2 / 414.0	—
MTP, 1 token (Quick start)	124.8 / 206.7 / 336.0 / 525.2	1.74
MTP, 2 tokens	111.4 / 181.9 / 301.6 / 468.0	2.17

MTP 1 is the fastest at every concurrency. At temperature 0 the MTP output is not identical to the run without speculation (verification changes the floating-point rounding).

Sampling and repetition loops

RTX 3090, KV fp8, MTP 1, --max-num-seqs 8, 2026-10-26: 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated word 8-grams; an empty answer = the reasoning used the whole budget.

Table with columns: Temperature, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.5	0.6	0.7	0.8	0.9	1.0
loops / 30	3	1	3	1	1	2
finished before 16,000 tokens / 30	29	30	29	30	29	30
empty answers / 30	1	0	0	0	0	0
mean tokens	6,643	6,201	6,787	6,400	6,911	7,538

Few loops at every temperature, but answers are long (6–7.5k tokens with the reasoning). The BF16 original was not run in this test. Use the authors' sampling (temperature 1.0, top_p 0.95, top_k 20), the default in generation_config.json.

Known model behaviour and discarded options

Recall without thinking and text instead of a tool call after a bare request: see Test battery; both happen with the BF16 original too.
Discarded: plain 4.00 bpw (twice the KLD) and -hq 2 (5–8 % lower KLD for 0.35 GB more and 6 % slower). See Quantization quality.
Discarded: calibration on our sysadmin conversations (−5 % / −6 % KLD on our traces, +3 % on the public one).
Discarded: MTP 2: higher acceptance but slower than MTP 1 (table above).
Not reached: 262,144 tokens on 16 GB (212,992 is the measured maximum).

Requirements & install

EXL3 is not a native vLLM format. Two plugins are loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc + patches/orcasaq2-7ddffb12.diff): EXL3 linear layers through exllamav3's kernels.
qwen38_exl3_vision (ours, plugin/qwen38_exl3_vision/): the EXL3 vision tower and MTP head.
exllamav3 1.5.1 (the serving runtime; the checkpoint was converted with 1.6.4) and Python 3.12 (the exllamav3 wheels are cp312).

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \  --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match   # FlashInfer kernelsW=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"hf download JRamirez-UAB/Groot-19B-A3B-EXL3-4.00bpw-16-24GB --include "plugin/*" "patches/*" --local-dir .git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc && git apply ../patches/orcasaq2-7ddffb12.diff)uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start

24 GB GPU, vision + MTP 1, 262,144-token context, up to 8 concurrent sequences, 1 image per request (≤1 MP):

bash

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Groot-19B-A3B-EXL3-4.00bpw-16-24GB \  --served-model-name groot-19b-a3b \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 11296602849 \  --max-num-batched-tokens 1024 \  --max-num-seqs 8 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

16 GB: --max-model-len 212992 --kv-cache-memory-bytes 2655707463.

Checked end to end: a new environment built with setup.sh (plugin 0.4.1), the 24 GB Quick start above served as written, and our battery run against it: smoke OK, tools 20/30, vision 14/15, needle 7/8 · 6/8 · 5/8 at 64k · 128k · 241k, in line with the campaign results.

Using it with opencode (how we tested)

Tested with opencode 1.18.35. Tool calling was measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run directly against the vLLM API.

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "groot-19b-a3b": {          "name": "Groot-19B-A3B EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 16000 }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/groot-19b-a3b", "temperature": 1.0 },    "plan":  { "model": "local-vllm/groot-19b-a3b", "temperature": 1.0 }  }}

Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model.
limit.context = --max-model-len (212992 on 16 GB). Without attachment / modalities opencode will not send images.

Important notes

Memory. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations: on the RTX 3090 a start at 97 % utilization ran out of memory in the warm-up). Read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv; a desktop session on the same GPU eats the margin in the capacity table.
VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 is required for this MoE with the EXL3 plugin.
Hugging Face page warnings: the "Model size" Hugging Face shows counts packed storage elements (EXL3 packs several weights into each int16), not parameters. The Configuration Parsing Warning "quantization_config.bits must be an integer" is expected: for EXL3 bits is an average.
VLLM_USE_FLASHINFER_SAMPLER=0 is recommended by the OrcaSAQ2 plugin. Use --tool-call-parser qwen3_coder and --reasoning-parser qwen3.

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Conversion	exllamav3 1.6.4 `convert_model`: 4.00 bpw (`-b 4.00`), `-hq 1` (attention and shared expert 6 bits), `lm_head` 6 bits (`-hb 6`), vision 6 bits (`-vb 6`), MTP 4 bits (`-mb 4`), embeddings BF16
Calibration	exllamav3's built-in calibration set, 250 rows × 2,048 tokens
Bit allocation	uniform rate on the routed experts, 6 bits on attention and the shared expert
Hardware	NVIDIA A40 + RTX PRO 6000 (`-d 0,1 -dr 1,2`)

Tokenizer, chat template, processor configs and generation_config.json unchanged.

Integrity and validation

Artifact integrity

Every file is checked against our local copy with SHA-256 (SHA256SUMS); the checkpoint was downloaded back from this repo, installed with setup.sh in a new environment and served with the 24 GB Quick start (RTX PRO 6000, 2026-10-26): smoke OK, tools 20/30, vision 14/15, needle 7/8 · 6/8 · 5/8.

Quantization

Fidelity: KL divergence against BF16 with exllamav3's qbench on three traces (table above).

Chat template

Unchanged from the base model; tool calls parse with qwen3_coder (34/60 in the replay of the real opencode request, all valid).

Runtime

Load, needle, tool calls, vision, agent tasks, concurrency, VRAM and the loop test: see Results; engine errors during the runs: 0.

Scope: results apply to this revision, the stacks above and the L40S / RTX 3090 / RTX PRO 6000 GPUs with the settings stated; other setups are not tested.

Limitations

Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine.
Needs our qwen38_exl3_vision plugin and the OrcaSAQ2 plugin; it will not load with stock vLLM, Transformers or SGLang.
Recall of facts without thinking and tool calling after short requests are weak (BF16 too).
16 GB tops out at 212,992 tokens; the 16 GB row was measured on an L40S with the KV cache and VRAM sized for it, not on a physical 16 GB card.
Vision: only single images (≤1 MP) were tested; not several images, video or photographs.
Quality was not evaluated with standard benchmarks.

Feedback welcome

I would be grateful for reports of installations of this checkpoint, both working and failed ones, especially on real 16 / 24 GB GPUs. Please open a discussion in the Community tab with your GPU, driver, vLLM / exllamav3 / plugin versions, the full command line and the result.

Credits & license

Base model: GrootTeam/Groot-19B-A3B-261023 (Apache-2.0), post-trained from Qwen3.6-35B-A3B.
exllamav3 and the EXL3 format by turboderp (MIT), including qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
Our code: plugin/qwen38_exl3_vision/, patches/, setup.sh (Apache-2.0).

Apache-2.0, same as the base model. This is a community quantization, not an official release of its authors. exllamav3, vLLM and OrcaSAQ2 are not redistributed here; install them from their repositories under their own licenses.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

GrootTeam/Groot-19B-A3B-261023

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What is in this repository

Which one to use

Table with columns: GPU size, Context per session, Full-context sessions, KV cache, Speculative decoding
GPU size	Context per session	Full-context sessions	KV cache	Speculative decoding
16 GB	212,992	1 (pool of 214,928 tokens)	fp8, 2,655,707,463 B	MTP 1
24 GB	262,144	3 (pool of 927,285 tokens)	fp8, 11,296,602,849 B	MTP 1

On 16 GB the full 262,144 tokens do not fit (the first attempt with a 3.83 GB KV cache went 759 MiB over).

Variants / revisions

Table with columns: Revision, Routed experts, Attention + shared expert, lm_head, Vision, MTP, Embeddings, Size, KLD (sysadmin / opencode / public)
Revision	Routed experts	Attention + shared expert	`lm_head`	Vision	MTP	Embeddings	Size	KLD (sysadmin / opencode / public)
`main`	4.00 bpw	6 bits	6 bits	6 bits	4 bits	BF16	11.5 GB	0.0108 / 0.0100 / 0.0105

Files and reproducibility

model-*.safetensors, config.json (quantization_config: exl3), quantization_config.json (per-tensor storage), tokenizer files, chat_template.jinja, the processor configs and generation_config.json (unchanged: temperature 1.0, top_p 0.95, top_k 20, the authors' recommended sampling).
plugin/qwen38_exl3_vision/: our vLLM plugin (EXL3 vision tower and MTP head for the Qwen3.5 / Qwen3.6 architectures).
patches/orcasaq2-7ddffb12.diff: the changes our tested OrcaSAQ2 has on top of commit 7ddffb12. setup.sh: the install below as one script.
LICENSE (Apache-2.0, from the base model), SHA256SUMS.
Not included: the BF16 original, exllamav3, vLLM and OrcaSAQ2 (installed from their repositories), our calibration and evaluation traces.

Results

Quantization quality

Table with columns: Trace, KLD mean, KLD median, KLD p90, PPL BF16, PPL EXL3
Trace	KLD mean	KLD median	KLD p90	PPL BF16	PPL EXL3
Our sysadmin trace	0.0108	0.0038	0.0276	3.5345	3.5407
opencode sessions	0.0100	0.0035	0.0251	4.3234	4.3219
turboderp's public trace	0.0105	0.0017	0.0290	2.3879	2.3912

How the conversion was chosen (all at 4.00 bpw on the experts, same KLD traces):

Table with columns: Conversion, Size, KLD sysadmin / opencode / public, Decode, 1 / 8 requests (no MTP, RTX PRO 6000)
Conversion	Size	KLD sysadmin / opencode / public	Decode, 1 / 8 requests (no MTP, RTX PRO 6000)
plain `-b 4.00`	11.1 GB	0.0212 / 0.0196 / 0.0203	116 / 618 tok/s
`-hq 1` (this repo)	11.5 GB	0.0108 / 0.0100 / 0.0105	108 / 590 tok/s
`-hq 2` (attention + shared expert at 8 bits)	11.8 GB	0.0101 / 0.0093 / 0.0100	101 / 583 tok/s
`-hq 1` + calibration on our sysadmin conversations	11.5 GB	0.0103 / 0.0094 / 0.0108	—

Capacity per GPU size

Table with columns: GPU, Size, --max-model-len, --kv-cache-memory-bytes, KV pool (tokens), Full-context sessions, Concurrent, VRAM peak, Really free at peak
GPU	Size	`--max-model-len`	`--kv-cache-memory-bytes`	KV pool (tokens)	Full-context sessions	Concurrent	VRAM peak	Really free at peak
L40S	16 GB	212,992	2,655,707,463	214,928	1	1	15,543 MiB	381 MiB
L40S	24 GB	262,144	11,296,602,849	927,285	3	4	23,713 MiB	403 MiB
RTX 3090 (real)	24 GB	262,144	11,337,035,939	931,198	3	4	23,450 MiB	~670 MiB

Test battery

Table with columns: Test, EXL3 4.00 -hq, BF16 original
Test	EXL3 4.00 `-hq`	BF16 original
Smoke (short answer; 4 long Spanish answers, 8,000-token limit)	all OK (3,222–4,627 tokens); no CJK (in the two end-to-end runs below, 2 CJK or full-width characters in 1 of 8 long answers)	—
Speed (decode, 1 / 2 / 4 / 8 requests)	139 / 249 / 435 / 716 tok/s	—
Tool calls, real opencode request	battery 17/30; replay 34/60 (20/20 full session, 2/20 "prompt + ls", 12/20 "greeting + story + ls")	battery 16/30; replay 32/60 (20 / 3 / 9)
Reasoning (48 problems)	40/48, 50,399 reasoning tokens (median 1,063), none cut	—
Vision (4 cases, 15 checks)	13/15	—
Needle (8 facts)	64k 7/8 · 128k 6/8 · 241k 5/8	64k 5/8 · 128k 5/8 · 241k 5/8
Needle with thinking, ~250k (3 runs: T 0, T 1.0, T 1.0 seed 7)	7/8 · 7/8 · 8/8 (the T 0 miss is one digit wrong: GEZ-5650 for GEZ-5660)	8/8 · 7/8 · 8/8
Concurrent sessions (8 × 30k)	0 of 8 returned all 8 facts (5–7 each)	0 of 8
Agent tasks (opencode, 10 sysadmin tasks × 3 runs)	7/10 · 7/10 · 7/10 = 21/30 (276–347 s per run)	—
Games	39/48 points, 1 of 4 run	41/48, 1 of 4 run (an earlier run: 43/48, 3 of 4)

Speculative decoding

RTX 3090 (24 GB, real), 262,144-token context, --max-num-seqs 8, KV fp8, vLLM 0.32.0rc1 + exllamav3 1.6.1 + FlashInfer.

Table with columns: Variant, 1 / 2 / 4 / 8 requests (tok/s), Acceptance
Variant	1 / 2 / 4 / 8 requests (tok/s)	Acceptance
none	96.6 / 156.8 / 263.2 / 414.0	—
MTP, 1 token (Quick start)	124.8 / 206.7 / 336.0 / 525.2	1.74
MTP, 2 tokens	111.4 / 181.9 / 301.6 / 468.0	2.17

MTP 1 is the fastest at every concurrency. At temperature 0 the MTP output is not identical to the run without speculation (verification changes the floating-point rounding).

Sampling and repetition loops

Table with columns: Temperature, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.5	0.6	0.7	0.8	0.9	1.0
loops / 30	3	1	3	1	1	2
finished before 16,000 tokens / 30	29	30	29	30	29	30
empty answers / 30	1	0	0	0	0	0
mean tokens	6,643	6,201	6,787	6,400	6,911	7,538

Known model behaviour and discarded options

Recall without thinking and text instead of a tool call after a bare request: see Test battery; both happen with the BF16 original too.
Discarded: plain 4.00 bpw (twice the KLD) and -hq 2 (5–8 % lower KLD for 0.35 GB more and 6 % slower). See Quantization quality.
Discarded: calibration on our sysadmin conversations (−5 % / −6 % KLD on our traces, +3 % on the public one).
Discarded: MTP 2: higher acceptance but slower than MTP 1 (table above).
Not reached: 262,144 tokens on 16 GB (212,992 is the measured maximum).

Requirements & install

EXL3 is not a native vLLM format. Two plugins are loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc + patches/orcasaq2-7ddffb12.diff): EXL3 linear layers through exllamav3's kernels.
qwen38_exl3_vision (ours, plugin/qwen38_exl3_vision/): the EXL3 vision tower and MTP head.
exllamav3 1.5.1 (the serving runtime; the checkpoint was converted with 1.6.4) and Python 3.12 (the exllamav3 wheels are cp312).

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \  --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match   # FlashInfer kernelsW=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"hf download JRamirez-UAB/Groot-19B-A3B-EXL3-4.00bpw-16-24GB --include "plugin/*" "patches/*" --local-dir .git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc && git apply ../patches/orcasaq2-7ddffb12.diff)uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \  --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match   # FlashInfer kernelsW=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"hf download JRamirez-UAB/Groot-19B-A3B-EXL3-4.00bpw-16-24GB --include "plugin/*" "patches/*" --local-dir .git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc && git apply ../patches/orcasaq2-7ddffb12.diff)uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start

24 GB GPU, vision + MTP 1, 262,144-token context, up to 8 concurrent sequences, 1 image per request (≤1 MP):

bash

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Groot-19B-A3B-EXL3-4.00bpw-16-24GB \  --served-model-name groot-19b-a3b \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 11296602849 \  --max-num-batched-tokens 1024 \  --max-num-seqs 8 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0 VLLM_DISABLE_SHARED_EXPERTS_STREAM=1vllm serve JRamirez-UAB/Groot-19B-A3B-EXL3-4.00bpw-16-24GB \  --served-model-name groot-19b-a3b \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 11296602849 \  --max-num-batched-tokens 1024 \  --max-num-seqs 8 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

16 GB: --max-model-len 212992 --kv-cache-memory-bytes 2655707463.

Using it with opencode (how we tested)

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "groot-19b-a3b": {          "name": "Groot-19B-A3B EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 16000 }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/groot-19b-a3b", "temperature": 1.0 },    "plan":  { "model": "local-vllm/groot-19b-a3b", "temperature": 1.0 }  }}

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "groot-19b-a3b": {          "name": "Groot-19B-A3B EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 16000 }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/groot-19b-a3b", "temperature": 1.0 },    "plan":  { "model": "local-vllm/groot-19b-a3b", "temperature": 1.0 }  }}

Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model.
limit.context = --max-model-len (212992 on 16 GB). Without attachment / modalities opencode will not send images.

Important notes

Memory. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations: on the RTX 3090 a start at 97 % utilization ran out of memory in the warm-up). Read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv; a desktop session on the same GPU eats the margin in the capacity table.
VLLM_DISABLE_SHARED_EXPERTS_STREAM=1 is required for this MoE with the EXL3 plugin.
Hugging Face page warnings: the "Model size" Hugging Face shows counts packed storage elements (EXL3 packs several weights into each int16), not parameters. The Configuration Parsing Warning "quantization_config.bits must be an integer" is expected: for EXL3 bits is an average.
VLLM_USE_FLASHINFER_SAMPLER=0 is recommended by the OrcaSAQ2 plugin. Use --tool-call-parser qwen3_coder and --reasoning-parser qwen3.

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Conversion	exllamav3 1.6.4 `convert_model`: 4.00 bpw (`-b 4.00`), `-hq 1` (attention and shared expert 6 bits), `lm_head` 6 bits (`-hb 6`), vision 6 bits (`-vb 6`), MTP 4 bits (`-mb 4`), embeddings BF16
Calibration	exllamav3's built-in calibration set, 250 rows × 2,048 tokens
Bit allocation	uniform rate on the routed experts, 6 bits on attention and the shared expert
Hardware	NVIDIA A40 + RTX PRO 6000 (`-d 0,1 -dr 1,2`)

Tokenizer, chat template, processor configs and generation_config.json unchanged.

Integrity and validation

Artifact integrity

Every file is checked against our local copy with SHA-256 (SHA256SUMS); the checkpoint was downloaded back from this repo, installed with setup.sh in a new environment and served with the 24 GB Quick start (RTX PRO 6000, 2026-10-26): smoke OK, tools 20/30, vision 14/15, needle 7/8 · 6/8 · 5/8.

Quantization

Fidelity: KL divergence against BF16 with exllamav3's qbench on three traces (table above).

Chat template

Unchanged from the base model; tool calls parse with qwen3_coder (34/60 in the replay of the real opencode request, all valid).

Runtime

Load, needle, tool calls, vision, agent tasks, concurrency, VRAM and the loop test: see Results; engine errors during the runs: 0.

Scope: results apply to this revision, the stacks above and the L40S / RTX 3090 / RTX PRO 6000 GPUs with the settings stated; other setups are not tested.

Limitations

Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine.
Needs our qwen38_exl3_vision plugin and the OrcaSAQ2 plugin; it will not load with stock vLLM, Transformers or SGLang.
Recall of facts without thinking and tool calling after short requests are weak (BF16 too).
16 GB tops out at 212,992 tokens; the 16 GB row was measured on an L40S with the KV cache and VRAM sized for it, not on a physical 16 GB card.
Vision: only single images (≤1 MP) were tested; not several images, video or photographs.
Quality was not evaluated with standard benchmarks.

Feedback welcome

Credits & license

Base model: GrootTeam/Groot-19B-A3B-261023 (Apache-2.0), post-trained from Qwen3.6-35B-A3B.
exllamav3 and the EXL3 format by turboderp (MIT), including qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
Our code: plugin/qwen38_exl3_vision/, patches/, setup.sh (Apache-2.0).