JRamirez-UAB

Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB

README

License: apache-2.0

What is in this repository

Which one to use

Table with columns: GPU size, Revision, Context per session, Full-context sessions, KV cache, Speculative decoding
GPU size	Revision	Context per session	Full-context sessions	KV cache	Speculative decoding
8 GB	`4.50bpw`	69,632	1	fp8, 1,496,317,952 B	MTP 1
12 GB	`main` (6.00)	262,144	1	fp8, 5,317,412,782 B	MTP 1
16 GB	`main` (6.00)	262,144	1 (pool of 500,918 tokens)	fp8, 9,596,441,722 B	MTP 1
16 GB, two full-context sessions	`4.50bpw`	262,144	2 (pool of 525,304 tokens)	fp8, 10,074,760,151 B	MTP 1

main is the better model wherever it fits (about 7× lower KLD). On 8 GB main only reaches a 44,032-token context, so 4.50bpw is the choice there. 5.37bpw (4.6 GB) is 0.27 GB smaller than main; its capacity per GPU size was not measured, so it is not in this table.

Variants / revisions

Table with columns: Revision, Language model, lm_head, Vision, MTP, Embeddings, Size, KLD (sysadmin / opencode / public)
Revision	Language model	`lm_head`	Vision	MTP	Embeddings	Size	KLD (sysadmin / opencode / public)
`main` (agent-session calibration)	6.00 bpw	6 bits	6 bits	4 bits	BF16	4.9 GB	0.0020 / 0.0021 / 0.0016
`5.37bpw` (per-layer bits, agent-session calibration)	5.37 bpw (5 bits; 6 bits on layers 2–11, 18, 19)	6 bits	6 bits	4 bits	BF16	4.6 GB	0.0041 / 0.0041 / 0.0031
`4.50bpw`	4.50 bpw	6 bits	6 bits	4 bits	BF16	4.2 GB	0.0143 / 0.0144 / 0.0109
`6.00bpw-standard-calibration` (previous `main`)	6.00 bpw	6 bits	6 bits	4 bits	BF16	4.9 GB	0.0024 / 0.0023 / 0.0018

bash

hf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --revision 4.50bpw --local-dir Kestrel-4B-Coder-EXL3-4.50bpwhf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --revision 5.37bpw --local-dir Kestrel-4B-Coder-EXL3-5.37bpw

The 6.00bpw-standard-calibration branch keeps the previous main byte for byte, for anyone who wants to keep using it; it has no advantage over the new main.

Files and reproducibility

model.safetensors, config.json (quantization_config: exl3, head 6, vision 6, MTP 4), quantization_config.json (per-tensor storage), tokenizer files (tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json), chat_template.jinja, the processor configs and generation_config.json (our recommended sampling, see Sampling and repetition loops).
plugin/qwen38_exl3_vision/: our vLLM plugin (EXL3 vision tower and MTP head for the Qwen3.5 / Qwen3.8 architecture).
patches/orcasaq2-7ddffb12.diff: the changes our tested OrcaSAQ2 has on top of commit 7ddffb12. setup.sh: the install below as one script.
LICENSE (Apache-2.0, from the base model), SHA256SUMS.
Not included: the BF16 original, exllamav3, vLLM and OrcaSAQ2 (installed from their repositories), our calibration and evaluation traces.

Results

Quantization quality

KL divergence and perplexity against the BF16 original with exllamav3's qbench (2026-10-25/27). Traces: our own Spanish sysadmin trace, a trace built from real opencode sessions, and turboderp's public qbench_prompts_gen.json. No session in the calibration of main is in these traces, but the opencode trace shares opencode's system prompt with it, which favours main there; the public trace shares nothing with it.

Table with columns: Trace, Revision, KLD mean, KLD median, KLD p90, PPL BF16, PPL EXL3
Trace	Revision	KLD mean	KLD median	KLD p90	PPL BF16	PPL EXL3
Our sysadmin trace	`main`	0.0020	0.0011	0.0052	2.7255	2.7291
	`6.00bpw-standard-calibration`	0.0024	0.0013	0.0061		2.7297
	`5.37bpw`	0.0041	0.0021	0.0104		2.7358
	`4.50bpw`	0.0143	0.0074	0.0365		2.7698
opencode sessions	`main`	0.0021	0.0010	0.0053	3.2382	3.2411
	`6.00bpw-standard-calibration`	0.0023	0.0011	0.0059		3.2422
	`5.37bpw`	0.0041	0.0019	0.0106		3.2479
	`4.50bpw`	0.0144	0.0067	0.0365		3.2665
turboderp's public trace	`main`	0.0016	0.0004	0.0045	2.0771	2.0792
	`6.00bpw-standard-calibration`	0.0018	0.0005	0.0051		2.0789
	`5.37bpw`	0.0031	0.0009	0.0089		2.0816
	`4.50bpw`	0.0109	0.0031	0.0308		2.1004

Capacity per GPU size

Measured with the KV cache sized for each GPU size (main on an A40, 4.50bpw on an L40S); the VRAM limit of each size assumes the 460 MiB a real GPU of that size reserves for the driver. Vision (1 image, ≤1 MP) and MTP 1 enabled, KV fp8, --max-num-seqs 4. "Pool" is the GPU KV cache size vLLM reports at start-up; "full-context sessions" = pool ÷ context, rounded down.

Table with columns: Revision, GPU size, --max-model-len, --kv-cache-memory-bytes, KV pool (tokens), Full-context sessions, VRAM peak, Really free at peak
Revision	GPU size	`--max-model-len`	`--kv-cache-memory-bytes`	KV pool (tokens)	Full-context sessions	VRAM peak	Really free at peak
`main`	8 GB	44,032	1,028,233,625	44,895	1	7,409 MiB	323 MiB
`main`	12 GB	262,144	5,317,412,782	277,384	1	11,475 MiB	353 MiB
`main`	16 GB	262,144	9,596,441,722	500,918	1	15,555 MiB	369 MiB
`4.50bpw`	8 GB	69,632	1,496,317,952	70,560	1	7,361 MiB	371 MiB
`4.50bpw`	12 GB	262,144	5,774,759,690	300,754	1	11,463 MiB	365 MiB
`4.50bpw`	16 GB	262,144	10,074,760,151	525,304	2	15,563 MiB	361 MiB

Tool calls and agent tasks of the current `main` (2026-10-27)

Same settings as the battery below. Each run of main was paired with a run of the standard-calibration 6.00 on the same GPU and day; four runs of that 6.00 in total. Run-to-run noise is large: the same 6.00 scored 51/60 in the replay on 2026-10-25 and 44–47/60 on 2026-10-27.

Table with columns: Test, main (A40), main (RTX PRO 6000), standard-calibration 6.00, same day (4 runs)
Test	`main` (A40)	`main` (RTX PRO 6000)	standard-calibration 6.00, same day (4 runs)
Tool calls, battery	28/30	28/30	25–27/30
Tool calls, replay of the real opencode request (3 cases × 20)	50/60	52/60	44–47/60
Vision (15 checks)	12/15	13/15	13–14/15
Agent tasks (10 tasks × 3 runs)	11/30	13/30	8–16/30

`5.37bpw`: per-layer bit allocation (2026-10-27)

How the layers were chosen: the conversion log of exllamav3 reports the relative error (rfn) of each layer. We converted a uniform 5-bit build and ranked its 32 layers; 4- and 6-bit conversions ranked them almost the same way. The 12 worst (2–11, 18 and 19) were raised to 6 bits with a per-tensor recipe (-rcp); no layer was lowered. After the change, the raised layers went from an rfn of 0.020–0.027 to 0.011–0.015, and layers 0, 1 and 12 became the worst (about 0.019). Both builds use the same agent-session calibration as main. Same settings as the battery below; each build had one run on an A40 and one on the RTX PRO 6000, with the BF16 original on the same GPUs and day.

Table with columns: 5.37bpw, uniform 5.00 bpw (not published), BF16 original, same day
	`5.37bpw`	uniform 5.00 bpw (not published)	BF16 original, same day
Size	4.6 GB	4.5 GB	—
KLD sysadmin / opencode / public	0.0041 / 0.0041 / 0.0031	0.0057 / 0.0058 / 0.0043	0
Tool calls, battery	28/30, 29/30	27/30, 29/30	26/30, 28/30
Tool calls, replay of the real opencode request	53/60, 50/60	42/60, 36/60	43/60, 39/60
Vision	13/15, 14/15	13/15, 13/15	13/15, 13/15
Agent tasks (10 tasks × 3 runs)	12/30, 17/30	15/30, 11/30	9/30, 13/30

How to read the replay: the BF16 original itself scores 39–43/60, and the 4.50 bpw build scores higher than BF16. The replay detects broken builds (they fall below 25/60), but it does not rank healthy ones; the KL divergence does. Agent tasks are noisy too: single runs of the same build range from 2/10 to 7/10.

Test battery

262,144-token context, KV fp8 with the 16 GB size of each revision, MTP 1, --max-num-seqs 4, temperature 1.0 (top_p 0.95, top_k 20), 2026-10-25/26. The 6.00 column is the standard-calibration 6.00 (now the 6.00bpw-standard-calibration branch); the current main has the same size and format, and its tool-call and agent results are in the table above. 6.00 on the A40, 4.50bpw on the RTX PRO 6000; the BF16 original on the RTX PRO 6000 with the same settings. With --max-num-seqs 4, 8 requests run as 4.

Table with columns: Test, standard-calibration 6.00, 4.50bpw, BF16 original
Test	standard-calibration 6.00	`4.50bpw`	BF16 original
Smoke (short answer; 4 long Spanish answers, 8,000-token limit)	short answer OK; 3 of 4 long answers finished, 1 reached the limit; no CJK	same: 3 of 4, no CJK	same: 3 of 4, no CJK
Speed (decode, 1 / 2 / 4 / 8 requests)	168 / 308 / 579 / 586 tok/s (A40)	335 / 525 / 1,029 / 1,033 tok/s (RTX PRO 6000)	204 / 387 / 755 / 764 tok/s (RTX PRO 6000)
Tool calls, real opencode request (3 cases × 20)	battery 28/30; replay 51/60 (all valid, 49 correct)	battery 29/30; replay 57/60 (all valid and correct)	battery 26/30; replay 48/60 (all valid and correct)
Reasoning (48 problems)	36/48, 31,110 reasoning tokens (median 440), 1 cut	38/48, 28,175 reasoning tokens (median 429), none cut	36/48, 41,895 reasoning tokens (median 524), 1 cut
Vision (4 cases, 15 checks)	13/15	15/15	13/15
Needle (8 facts)	64k 8/8 · 128k 8/8 · 241k 8/8	64k 7/8 · 128k 8/8 · 241k 8/8	64k 8/8 · 128k 8/8 · 241k 8/8
Concurrent sessions (8 × 30k)	7 of 8 returned all 8 facts	8 of 8 returned all 8 facts	8 of 8 returned all 8 facts
Agent tasks (opencode, 10 sysadmin tasks × 3 runs)	5/10 · 5/10 · 4/10 = 14/30	4/10 · 6/10 · 4/10 = 14/30	7/10 · 4/10 · 5/10 = 16/30
Games	39/48 points, 0 of 4 run	40/48 points, 0 of 4 run	42/48 points, 2 of 4 run

The chat template has no reasoning-effort setting (only enable_thinking): the xhigh the battery sends is ignored.

Speculative decoding

A40, 262,144-token context, --max-num-seqs 8, KV fp8.

Table with columns: Variant, main: 1 / 2 / 4 / 8 requests (tok/s), Acceptance, 4.50bpw: 1 / 2 / 4 / 8 requests (tok/s), Acceptance
Variant	`main`: 1 / 2 / 4 / 8 requests (tok/s)	Acceptance	`4.50bpw`: 1 / 2 / 4 / 8 requests (tok/s)	Acceptance
none	119.5 / 222.1 / 424.3 / 767.9	—	143.6 / 245.7 / 469.4 / 846.7	—
MTP, 1 token (Quick start)	166.9 / 326.8 / 574.5 / 1,026.3	1.85	171.0 / 312.1 / 614.5 / 1,066.6	1.82
MTP, 3 tokens	162.9 / 296.3 / 588.1 / 737.9	2.84	177.3 / 302.6 / 549.3 / 768.5	2.97

MTP 1 is clearly the fastest with 8 requests and within 6 % of MTP 3 elsewhere. At temperature 0 the MTP output is not identical to the run without speculation (verification changes the floating-point rounding); the battery and the loop test ran with MTP 1.

Sampling and repetition loops

L40S, KV fp8, MTP 1, --max-num-seqs 8, 2026-10-25: 5 long Spanish sysadmin prompts × 6 runs per temperature, up to 16,000 tokens, top_p 0.95 and top_k 20; a loop = more than 30 % repeated word 8-grams; an empty answer = the reasoning used the whole budget. The main rows here, in Speculative decoding and in Known model behaviour were measured with the standard-calibration 6.00 (same size and format as the current main; not re-run). The 5.37bpw rows were measured on 2026-10-27 on an A40 with the copy downloaded from this repo.

Table with columns: Temperature, 0.2, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.2	0.5	0.6	0.7	0.8	0.9	1.0
`main`: loops / 30	13	12	10	8	7	7	1
`main`: finished before 16,000 tokens / 30	13	17	19	19	23	21	26
`main`: empty answers / 30	1	1	2	0	1	3	2
`5.37bpw`: loops / 30	–	11	11	8	9	3	1
`5.37bpw`: finished / 30	–	17	19	21	18	23	22
`5.37bpw`: empty / 30	–	4	4	1	5	3	5
`4.50bpw`: loops / 30	22	14	12	10	12	11	3
`4.50bpw`: finished / 30	6	13	14	19	17	17	26
`4.50bpw`: empty / 30	1	0	2	1	0	1	0
BF16 original: loops / 30	–	7	11	11	11	7	1
BF16 original: finished / 30	–	19	17	16	16	22	26
BF16 original: empty / 30	–	2	1	2	5	4	1

The BF16 original loops about as often as main below temperature 1.0, so the loops come from the model, not from the quantization.

Use temperature 1.0: it is this repo's default in generation_config.json, with top_p 0.95 and top_k 20. The author recommends temperature 0.2 for coding (short HumanEval answers); on our long sysadmin answers 0.2 loops in 13/30 (main) and 22/30 (4.50bpw).

Known model behaviour and discarded options

Chinese characters come from sampling settings, not from the quantization. The original generation_config.json sets no top_p / top_k; with it, vLLM samples the whole vocabulary at temperature 1.0 and Chinese characters appear (BF16: 4 of 10 English answers in our CJK test). With top_p 0.95 and top_k 20 (this repo's generation_config.json): BF16 0/40, main 1/40, 4.50bpw 0/40 answers with CJK characters.
Do not add min_p. vLLM 0.31 rejects every request that carries min_p when MTP speculative decoding is on (HTTP 400). With a local vLLM patch that makes it work, min_p 0.05 multiplied the loops of main: 17/60 at temperature 1.0 against 3/60 without it, and 29/60 against 7/60 at 0.9 (two runs of 30 each, 2026-10-27). It cuts the low-probability tail, which on this model is what keeps long answers from looping.
Do not use temperature 0 (greedy decoding). With a long sysadmin question in Spanish (3,000 max tokens) both main and the BF16 original loop in the answer at temperature 0 and never finish (repeated word 8-grams: 0.79 main, 0.63 BF16). It is the model, not the quantization. Use the sampling in generation_config.json (temperature 1.0, top-p 0.95, top-k 20); with it the same question does not loop.
Discarded: temperature 0.2 (the author's coding setting): more loops on long answers (table above).
Discarded: MTP 3: higher acceptance but slower than MTP 1 with 8 requests (table above).
Discarded on 8 GB: main (44,032-token context against 69,632 with 4.50bpw).

Requirements & install

EXL3 is not a native vLLM format. Two plugins are loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc + patches/orcasaq2-7ddffb12.diff): EXL3 linear layers through exllamav3's kernels.
qwen38_exl3_vision 0.4.1 (ours, plugin/qwen38_exl3_vision/): the EXL3 vision tower and MTP head.
exllamav3 1.5.1 (the serving runtime; the checkpoint was converted with 1.6.4) and Python 3.12 (the exllamav3 wheels are cp312).

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \  --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match   # FlashInfer kernelsW=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"hf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --include "plugin/*" "patches/*" --local-dir .git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc && git apply ../patches/orcasaq2-7ddffb12.diff)uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start

12 GB GPU, main, vision + MTP 1, 262,144-token context, up to 4 concurrent sequences, 1 image per request (≤1 MP):

bash

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB \  --served-model-name kestrel-4b-coder \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 5317412782 \  --max-num-batched-tokens 1024 \  --max-num-seqs 4 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

16 GB, main: --kv-cache-memory-bytes 9596441722 (one 262,144-token session, or several shorter ones sharing a 500,918-token pool).
16 GB, two full-context sessions: --revision 4.50bpw --kv-cache-memory-bytes 10074760151.
8 GB: --revision 4.50bpw --max-model-len 69632 --kv-cache-memory-bytes 1496317952.

Using it with opencode (how we tested)

Tested with opencode 1.18.35. Tool calling was measured with the real opencode request (its long system prompt and 11 tools, captured and replayed against vLLM) and with opencode run agent tasks; context, speed, memory and image tests were run directly against the vLLM API.

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "kestrel-4b-coder": {          "name": "Kestrel-4B-Coder EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 16000 }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/kestrel-4b-coder", "temperature": 1.0 },    "plan":  { "model": "local-vllm/kestrel-4b-coder", "temperature": 1.0 }  }}

Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model.
limit.context = --max-model-len (69632 on 8 GB). Without attachment / modalities opencode will not send images.

Important notes

Memory. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). Read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv; a desktop session on the same GPU eats the margin in the capacity table.
Hugging Face page warnings: the "Model size" Hugging Face shows counts packed storage elements (EXL3 packs several weights into each int16), not parameters. The Configuration Parsing Warning "quantization_config.bits must be an integer" is expected: for EXL3 bits is an average.
VLLM_USE_FLASHINFER_SAMPLER=0 is recommended by the OrcaSAQ2 plugin. Use --tool-call-parser qwen3_coder and --reasoning-parser qwen3.

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Conversion	exllamav3 1.6.4 `convert_model`: 6.00 bpw (`-b 6`), 5.37 bpw (`-b 5` plus a per-tensor recipe with `-rcp`) and 4.50 bpw (`-b 4.5`), `lm_head` 6 bits, vision 6 bits (`-vb 6`), MTP 4 bits (`-mb 4`), embeddings BF16, codebook `mul1`
Calibration of `main` and `5.37bpw` (2026-10-27)	590 rows, about 1.15 M tokens: exllamav3's built-in set (250 rows × 2,048 tokens: web, code, multilingual, wiki) first, then real sessions of our Linux system-administration agent (opencode with tool calling, mostly in Spanish) from a university IT department, through the model's chat template with opencode's real system prompt and 11 tools. 400 conversations from a pool of about 2,600 sessions: 200 real session prefixes, with the real outputs of the commands the agent ran, each cut before its first failed tool call; and 200 next steps written by the BF16 original from real session contexts (reasoning and tool call; nothing was executed). The conversation parts are packed into 336 rows of 2,048 tokens, and 4 full sessions with the system prompt (up to 46,793 tokens) are added. Sessions in our evaluation traces were excluded. The data stays on our servers; the weights only carry the resulting quantization, and `quantization_config.json` records only the row and column counts
Calibration of `4.50bpw` and `6.00bpw-standard-calibration`	exllamav3's built-in calibration set, 250 rows × 2,048 tokens
Bit allocation	`main`, `4.50bpw`, `6.00bpw-standard-calibration`: uniform target rate (no SC per-tensor optimisation). `5.37bpw`: 5 bits, with every linear tensor of layers 2–11, 18 and 19 at 6 bits (76 tensors at 6 bits, 124 at 5), chosen by the per-layer error of the uniform 5-bit conversion
Hardware	one A40

Tokenizer, chat template and processor configs unchanged; generation_config.json changed to our recommended sampling (temperature 1.0, top_p 0.95, top_k 20).

Integrity and validation

Artifact integrity

Every file is checked against our local copy with SHA-256 (SHA256SUMS); the checkpoint was downloaded back from this repo, installed with setup.sh in a new environment and served with the Quick start.

Quantization

Fidelity: KL divergence against BF16 with exllamav3's qbench on three traces (table above).

Chat template

Unchanged from the base model; tool calls parse with qwen3_coder (main 50/60 and 52/60, 5.37bpw 53/60 and 50/60, 4.50bpw 57/60 in the replay of the real opencode request).

Runtime

Load, needle, tool calls, vision, agent tasks, concurrency, VRAM and the loop test: see Results; engine errors during the runs: 0.

Scope: results apply to these revisions, the stack above (vLLM 0.31.0 + exllamav3 1.5.1 + OrcaSAQ2 7ddffb12 with our patch + qwen38_exl3_vision), the A40 / L40S / RTX PRO 6000 GPUs and the settings stated above; other setups are not tested.

Limitations

Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine.
Needs our qwen38_exl3_vision plugin and the OrcaSAQ2 plugin; it will not load with stock vLLM, Transformers or SGLang.
Agent tasks: 11–14/30 depending on revision and run (the BF16 original: 16/30), clearly below the 27B models we publish; use it where memory is the constraint.
The calibration of main is tuned to our use (Spanish sysadmin agent with opencode). The public trace improved as much as ours, but other domains were not tested separately.
5.37bpw: capacity per GPU size, speed and the needle test were not measured for it; only KL divergence, the battery above and the loop test.
The 8 / 12 / 16 GB rows were measured on 48 GB GPUs with the KV cache and VRAM sized for each size, not on physical 8 / 12 / 16 GB cards.
Vision: only single images (≤1 MP) were tested; not several images, video or photographs.
Quality was not evaluated with standard benchmarks. All numbers come from one software stack.

Feedback welcome

I would be grateful for reports of installations of this checkpoint, both working and failed ones, especially on real 8 / 12 / 16 GB GPUs. Please open a discussion in the Community tab with your GPU, driver, vLLM / exllamav3 / plugin versions, the full command line and the result.

Credits & license

Base model: Jackrong/Kestrel-4B-Coder (Apache-2.0), a fine-tune of Qwen3.5-4B.
exllamav3 and the EXL3 format by turboderp (MIT), including qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
Our code: plugin/qwen38_exl3_vision/, patches/, setup.sh (Apache-2.0).

Apache-2.0, same as the base model. This is a community quantization, not an official release of its author. exllamav3, vLLM and OrcaSAQ2 are not redistributed here; install them from their repositories under their own licenses.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

JRamirez-UAB

Model Tree

Base

Jackrong/Kestrel-4B-Coder

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

What is in this repository

Which one to use

Table with columns: GPU size, Revision, Context per session, Full-context sessions, KV cache, Speculative decoding
GPU size	Revision	Context per session	Full-context sessions	KV cache	Speculative decoding
8 GB	`4.50bpw`	69,632	1	fp8, 1,496,317,952 B	MTP 1
12 GB	`main` (6.00)	262,144	1	fp8, 5,317,412,782 B	MTP 1
16 GB	`main` (6.00)	262,144	1 (pool of 500,918 tokens)	fp8, 9,596,441,722 B	MTP 1
16 GB, two full-context sessions	`4.50bpw`	262,144	2 (pool of 525,304 tokens)	fp8, 10,074,760,151 B	MTP 1

Variants / revisions

Table with columns: Revision, Language model, lm_head, Vision, MTP, Embeddings, Size, KLD (sysadmin / opencode / public)
Revision	Language model	`lm_head`	Vision	MTP	Embeddings	Size	KLD (sysadmin / opencode / public)
`main` (agent-session calibration)	6.00 bpw	6 bits	6 bits	4 bits	BF16	4.9 GB	0.0020 / 0.0021 / 0.0016
`5.37bpw` (per-layer bits, agent-session calibration)	5.37 bpw (5 bits; 6 bits on layers 2–11, 18, 19)	6 bits	6 bits	4 bits	BF16	4.6 GB	0.0041 / 0.0041 / 0.0031
`4.50bpw`	4.50 bpw	6 bits	6 bits	4 bits	BF16	4.2 GB	0.0143 / 0.0144 / 0.0109
`6.00bpw-standard-calibration` (previous `main`)	6.00 bpw	6 bits	6 bits	4 bits	BF16	4.9 GB	0.0024 / 0.0023 / 0.0018

bash

hf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --revision 4.50bpw --local-dir Kestrel-4B-Coder-EXL3-4.50bpwhf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --revision 5.37bpw --local-dir Kestrel-4B-Coder-EXL3-5.37bpw

hf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --revision 4.50bpw --local-dir Kestrel-4B-Coder-EXL3-4.50bpwhf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --revision 5.37bpw --local-dir Kestrel-4B-Coder-EXL3-5.37bpw

The 6.00bpw-standard-calibration branch keeps the previous main byte for byte, for anyone who wants to keep using it; it has no advantage over the new main.

Files and reproducibility

model.safetensors, config.json (quantization_config: exl3, head 6, vision 6, MTP 4), quantization_config.json (per-tensor storage), tokenizer files (tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json), chat_template.jinja, the processor configs and generation_config.json (our recommended sampling, see Sampling and repetition loops).
plugin/qwen38_exl3_vision/: our vLLM plugin (EXL3 vision tower and MTP head for the Qwen3.5 / Qwen3.8 architecture).
patches/orcasaq2-7ddffb12.diff: the changes our tested OrcaSAQ2 has on top of commit 7ddffb12. setup.sh: the install below as one script.
LICENSE (Apache-2.0, from the base model), SHA256SUMS.
Not included: the BF16 original, exllamav3, vLLM and OrcaSAQ2 (installed from their repositories), our calibration and evaluation traces.

Results

Quantization quality

Table with columns: Trace, Revision, KLD mean, KLD median, KLD p90, PPL BF16, PPL EXL3
Trace	Revision	KLD mean	KLD median	KLD p90	PPL BF16	PPL EXL3
Our sysadmin trace	`main`	0.0020	0.0011	0.0052	2.7255	2.7291
	`6.00bpw-standard-calibration`	0.0024	0.0013	0.0061		2.7297
	`5.37bpw`	0.0041	0.0021	0.0104		2.7358
	`4.50bpw`	0.0143	0.0074	0.0365		2.7698
opencode sessions	`main`	0.0021	0.0010	0.0053	3.2382	3.2411
	`6.00bpw-standard-calibration`	0.0023	0.0011	0.0059		3.2422
	`5.37bpw`	0.0041	0.0019	0.0106		3.2479
	`4.50bpw`	0.0144	0.0067	0.0365		3.2665
turboderp's public trace	`main`	0.0016	0.0004	0.0045	2.0771	2.0792
	`6.00bpw-standard-calibration`	0.0018	0.0005	0.0051		2.0789
	`5.37bpw`	0.0031	0.0009	0.0089		2.0816
	`4.50bpw`	0.0109	0.0031	0.0308		2.1004

Capacity per GPU size

Table with columns: Revision, GPU size, --max-model-len, --kv-cache-memory-bytes, KV pool (tokens), Full-context sessions, VRAM peak, Really free at peak
Revision	GPU size	`--max-model-len`	`--kv-cache-memory-bytes`	KV pool (tokens)	Full-context sessions	VRAM peak	Really free at peak
`main`	8 GB	44,032	1,028,233,625	44,895	1	7,409 MiB	323 MiB
`main`	12 GB	262,144	5,317,412,782	277,384	1	11,475 MiB	353 MiB
`main`	16 GB	262,144	9,596,441,722	500,918	1	15,555 MiB	369 MiB
`4.50bpw`	8 GB	69,632	1,496,317,952	70,560	1	7,361 MiB	371 MiB
`4.50bpw`	12 GB	262,144	5,774,759,690	300,754	1	11,463 MiB	365 MiB
`4.50bpw`	16 GB	262,144	10,074,760,151	525,304	2	15,563 MiB	361 MiB

Tool calls and agent tasks of the current `main` (2026-10-27)

Table with columns: Test, main (A40), main (RTX PRO 6000), standard-calibration 6.00, same day (4 runs)
Test	`main` (A40)	`main` (RTX PRO 6000)	standard-calibration 6.00, same day (4 runs)
Tool calls, battery	28/30	28/30	25–27/30
Tool calls, replay of the real opencode request (3 cases × 20)	50/60	52/60	44–47/60
Vision (15 checks)	12/15	13/15	13–14/15
Agent tasks (10 tasks × 3 runs)	11/30	13/30	8–16/30

`5.37bpw`: per-layer bit allocation (2026-10-27)

Table with columns: 5.37bpw, uniform 5.00 bpw (not published), BF16 original, same day
	`5.37bpw`	uniform 5.00 bpw (not published)	BF16 original, same day
Size	4.6 GB	4.5 GB	—
KLD sysadmin / opencode / public	0.0041 / 0.0041 / 0.0031	0.0057 / 0.0058 / 0.0043	0
Tool calls, battery	28/30, 29/30	27/30, 29/30	26/30, 28/30
Tool calls, replay of the real opencode request	53/60, 50/60	42/60, 36/60	43/60, 39/60
Vision	13/15, 14/15	13/15, 13/15	13/15, 13/15
Agent tasks (10 tasks × 3 runs)	12/30, 17/30	15/30, 11/30	9/30, 13/30

Test battery

Table with columns: Test, standard-calibration 6.00, 4.50bpw, BF16 original
Test	standard-calibration 6.00	`4.50bpw`	BF16 original
Smoke (short answer; 4 long Spanish answers, 8,000-token limit)	short answer OK; 3 of 4 long answers finished, 1 reached the limit; no CJK	same: 3 of 4, no CJK	same: 3 of 4, no CJK
Speed (decode, 1 / 2 / 4 / 8 requests)	168 / 308 / 579 / 586 tok/s (A40)	335 / 525 / 1,029 / 1,033 tok/s (RTX PRO 6000)	204 / 387 / 755 / 764 tok/s (RTX PRO 6000)
Tool calls, real opencode request (3 cases × 20)	battery 28/30; replay 51/60 (all valid, 49 correct)	battery 29/30; replay 57/60 (all valid and correct)	battery 26/30; replay 48/60 (all valid and correct)
Reasoning (48 problems)	36/48, 31,110 reasoning tokens (median 440), 1 cut	38/48, 28,175 reasoning tokens (median 429), none cut	36/48, 41,895 reasoning tokens (median 524), 1 cut
Vision (4 cases, 15 checks)	13/15	15/15	13/15
Needle (8 facts)	64k 8/8 · 128k 8/8 · 241k 8/8	64k 7/8 · 128k 8/8 · 241k 8/8	64k 8/8 · 128k 8/8 · 241k 8/8
Concurrent sessions (8 × 30k)	7 of 8 returned all 8 facts	8 of 8 returned all 8 facts	8 of 8 returned all 8 facts
Agent tasks (opencode, 10 sysadmin tasks × 3 runs)	5/10 · 5/10 · 4/10 = 14/30	4/10 · 6/10 · 4/10 = 14/30	7/10 · 4/10 · 5/10 = 16/30
Games	39/48 points, 0 of 4 run	40/48 points, 0 of 4 run	42/48 points, 2 of 4 run

The chat template has no reasoning-effort setting (only enable_thinking): the xhigh the battery sends is ignored.

Speculative decoding

A40, 262,144-token context, --max-num-seqs 8, KV fp8.

Table with columns: Variant, main: 1 / 2 / 4 / 8 requests (tok/s), Acceptance, 4.50bpw: 1 / 2 / 4 / 8 requests (tok/s), Acceptance
Variant	`main`: 1 / 2 / 4 / 8 requests (tok/s)	Acceptance	`4.50bpw`: 1 / 2 / 4 / 8 requests (tok/s)	Acceptance
none	119.5 / 222.1 / 424.3 / 767.9	—	143.6 / 245.7 / 469.4 / 846.7	—
MTP, 1 token (Quick start)	166.9 / 326.8 / 574.5 / 1,026.3	1.85	171.0 / 312.1 / 614.5 / 1,066.6	1.82
MTP, 3 tokens	162.9 / 296.3 / 588.1 / 737.9	2.84	177.3 / 302.6 / 549.3 / 768.5	2.97

Sampling and repetition loops

Table with columns: Temperature, 0.2, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0
Temperature	0.2	0.5	0.6	0.7	0.8	0.9	1.0
`main`: loops / 30	13	12	10	8	7	7	1
`main`: finished before 16,000 tokens / 30	13	17	19	19	23	21	26
`main`: empty answers / 30	1	1	2	0	1	3	2
`5.37bpw`: loops / 30	–	11	11	8	9	3	1
`5.37bpw`: finished / 30	–	17	19	21	18	23	22
`5.37bpw`: empty / 30	–	4	4	1	5	3	5
`4.50bpw`: loops / 30	22	14	12	10	12	11	3
`4.50bpw`: finished / 30	6	13	14	19	17	17	26
`4.50bpw`: empty / 30	1	0	2	1	0	1	0
BF16 original: loops / 30	–	7	11	11	11	7	1
BF16 original: finished / 30	–	19	17	16	16	22	26
BF16 original: empty / 30	–	2	1	2	5	4	1

The BF16 original loops about as often as main below temperature 1.0, so the loops come from the model, not from the quantization.

Known model behaviour and discarded options

Chinese characters come from sampling settings, not from the quantization. The original generation_config.json sets no top_p / top_k; with it, vLLM samples the whole vocabulary at temperature 1.0 and Chinese characters appear (BF16: 4 of 10 English answers in our CJK test). With top_p 0.95 and top_k 20 (this repo's generation_config.json): BF16 0/40, main 1/40, 4.50bpw 0/40 answers with CJK characters.
Do not add min_p. vLLM 0.31 rejects every request that carries min_p when MTP speculative decoding is on (HTTP 400). With a local vLLM patch that makes it work, min_p 0.05 multiplied the loops of main: 17/60 at temperature 1.0 against 3/60 without it, and 29/60 against 7/60 at 0.9 (two runs of 30 each, 2026-10-27). It cuts the low-probability tail, which on this model is what keeps long answers from looping.
Do not use temperature 0 (greedy decoding). With a long sysadmin question in Spanish (3,000 max tokens) both main and the BF16 original loop in the answer at temperature 0 and never finish (repeated word 8-grams: 0.79 main, 0.63 BF16). It is the model, not the quantization. Use the sampling in generation_config.json (temperature 1.0, top-p 0.95, top-k 20); with it the same question does not loop.
Discarded: temperature 0.2 (the author's coding setting): more loops on long answers (table above).
Discarded: MTP 3: higher acceptance but slower than MTP 1 with 8 requests (table above).
Discarded on 8 GB: main (44,032-token context against 69,632 with 4.50bpw).

Requirements & install

EXL3 is not a native vLLM format. Two plugins are loaded through vLLM's vllm.general_plugins entry point:

Continuum-AI-Corp/OrcaSAQ2-kernel (commit 7ddffb12c00c24b8bc6b1164723c04ac490a91bc + patches/orcasaq2-7ddffb12.diff): EXL3 linear layers through exllamav3's kernels.
qwen38_exl3_vision 0.4.1 (ours, plugin/qwen38_exl3_vision/): the EXL3 vision tower and MTP head.
exllamav3 1.5.1 (the serving runtime; the checkpoint was converted with 1.6.4) and Python 3.12 (the exllamav3 wheels are cp312).

bash

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \  --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match   # FlashInfer kernelsW=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"hf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --include "plugin/*" "patches/*" --local-dir .git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc && git apply ../patches/orcasaq2-7ddffb12.diff)uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

uv venv --python 3.12 venvuv pip install --python venv/bin/python vllm==0.31.0 --torch-backend=auto        # torch 2.13.0uv pip install --python venv/bin/python flashinfer-cubin==0.7.0.post1 flashinfer-jit-cache==0.7.0.post1+cu130 \  --extra-index-url https://flashinfer.ai/whl/ --extra-index-url https://flashinfer.ai/whl/cu130/ --index-strategy unsafe-best-match   # FlashInfer kernelsW=exllamav3-1.5.1+cu132.torch2.13.0-cp312-cp312-linux_x86_64.whlcurl -L -o /tmp/$W "https://github.com/turboderp-org/exllamav3/releases/download/v1.5.1/${W/+/%2B}"hf download JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB --include "plugin/*" "patches/*" --local-dir .git clone https://github.com/Continuum-AI-Corp/OrcaSAQ2-kernel && (cd OrcaSAQ2-kernel && git checkout 7ddffb12c00c24b8bc6b1164723c04ac490a91bc && git apply ../patches/orcasaq2-7ddffb12.diff)uv pip install --python venv/bin/python /tmp/$W -e OrcaSAQ2-kernel -e plugin/qwen38_exl3_visionvenv/bin/python -c "import torch; from exllamav3.ext import exllamav3_ext; print(torch.__version__, torch.cuda.get_device_name(0))"

Quick start

12 GB GPU, main, vision + MTP 1, 262,144-token context, up to 4 concurrent sequences, 1 image per request (≤1 MP):

bash

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB \  --served-model-name kestrel-4b-coder \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 5317412782 \  --max-num-batched-tokens 1024 \  --max-num-seqs 4 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True VLLM_USE_FLASHINFER_SAMPLER=0vllm serve JRamirez-UAB/Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB \  --served-model-name kestrel-4b-coder \  --limit-mm-per-prompt '{"image":1,"video":0}' \  --mm-processor-kwargs '{"max_pixels":1003520}' \  --max-model-len 262144 \  --kv-cache-dtype fp8_e4m3 \  --kv-cache-memory-bytes 5317412782 \  --max-num-batched-tokens 1024 \  --max-num-seqs 4 \  --enable-prefix-caching \  --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \  --reasoning-parser qwen3 \  --enable-auto-tool-choice --tool-call-parser qwen3_coder

16 GB, main: --kv-cache-memory-bytes 9596441722 (one 262,144-token session, or several shorter ones sharing a 500,918-token pool).
16 GB, two full-context sessions: --revision 4.50bpw --kv-cache-memory-bytes 10074760151.
8 GB: --revision 4.50bpw --max-model-len 69632 --kv-cache-memory-bytes 1496317952.

Using it with opencode (how we tested)

json

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "kestrel-4b-coder": {          "name": "Kestrel-4B-Coder EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 16000 }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/kestrel-4b-coder", "temperature": 1.0 },    "plan":  { "model": "local-vllm/kestrel-4b-coder", "temperature": 1.0 }  }}

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "local-vllm": {      "npm": "@ai-sdk/openai-compatible",      "name": "local vLLM",      "options": { "baseURL": "http://127.0.0.1:8000/v1" },      "models": {        "kestrel-4b-coder": {          "name": "Kestrel-4B-Coder EXL3 (local)",          "tool_call": true,          "temperature": true,          "attachment": true,          "modalities": { "input": ["text", "image"], "output": ["text"] },          "limit": { "context": 262144, "output": 16000 }        }      }    }  },  "agent": {    "build": { "model": "local-vllm/kestrel-4b-coder", "temperature": 1.0 },    "plan":  { "model": "local-vllm/kestrel-4b-coder", "temperature": 1.0 }  }}

Declare "temperature": true in the model, otherwise opencode sends no temperature to a custom model.
limit.context = --max-model-len (69632 on 8 GB). Without attachment / modalities opencode will not send images.

Important notes

Memory. Size the KV cache with --kv-cache-memory-bytes, never with --gpu-memory-utilization (the profiling of this hybrid GatedDeltaNet model under-estimates the activations). Read the real headroom with nvidia-smi --query-gpu=memory.used,memory.free --format=csv; a desktop session on the same GPU eats the margin in the capacity table.
Hugging Face page warnings: the "Model size" Hugging Face shows counts packed storage elements (EXL3 packs several weights into each int16), not parameters. The Configuration Parsing Warning "quantization_config.bits must be an integer" is expected: for EXL3 bits is an average.
VLLM_USE_FLASHINFER_SAMPLER=0 is recommended by the OrcaSAQ2 plugin. Use --tool-call-parser qwen3_coder and --reasoning-parser qwen3.

Quantization recipe

Table with columns: Step, What we did
Step	What we did
Conversion	exllamav3 1.6.4 `convert_model`: 6.00 bpw (`-b 6`), 5.37 bpw (`-b 5` plus a per-tensor recipe with `-rcp`) and 4.50 bpw (`-b 4.5`), `lm_head` 6 bits, vision 6 bits (`-vb 6`), MTP 4 bits (`-mb 4`), embeddings BF16, codebook `mul1`
Calibration of `main` and `5.37bpw` (2026-10-27)	590 rows, about 1.15 M tokens: exllamav3's built-in set (250 rows × 2,048 tokens: web, code, multilingual, wiki) first, then real sessions of our Linux system-administration agent (opencode with tool calling, mostly in Spanish) from a university IT department, through the model's chat template with opencode's real system prompt and 11 tools. 400 conversations from a pool of about 2,600 sessions: 200 real session prefixes, with the real outputs of the commands the agent ran, each cut before its first failed tool call; and 200 next steps written by the BF16 original from real session contexts (reasoning and tool call; nothing was executed). The conversation parts are packed into 336 rows of 2,048 tokens, and 4 full sessions with the system prompt (up to 46,793 tokens) are added. Sessions in our evaluation traces were excluded. The data stays on our servers; the weights only carry the resulting quantization, and `quantization_config.json` records only the row and column counts
Calibration of `4.50bpw` and `6.00bpw-standard-calibration`	exllamav3's built-in calibration set, 250 rows × 2,048 tokens
Bit allocation	`main`, `4.50bpw`, `6.00bpw-standard-calibration`: uniform target rate (no SC per-tensor optimisation). `5.37bpw`: 5 bits, with every linear tensor of layers 2–11, 18 and 19 at 6 bits (76 tensors at 6 bits, 124 at 5), chosen by the per-layer error of the uniform 5-bit conversion
Hardware	one A40

Tokenizer, chat template and processor configs unchanged; generation_config.json changed to our recommended sampling (temperature 1.0, top_p 0.95, top_k 20).

Integrity and validation

Artifact integrity

Every file is checked against our local copy with SHA-256 (SHA256SUMS); the checkpoint was downloaded back from this repo, installed with setup.sh in a new environment and served with the Quick start.

Quantization

Fidelity: KL divergence against BF16 with exllamav3's qbench on three traces (table above).

Chat template

Unchanged from the base model; tool calls parse with qwen3_coder (main 50/60 and 52/60, 5.37bpw 53/60 and 50/60, 4.50bpw 57/60 in the replay of the real opencode request).

Runtime

Load, needle, tool calls, vision, agent tasks, concurrency, VRAM and the loop test: see Results; engine errors during the runs: 0.

Limitations

Review the commands before they run. As an agent, the model proposes shell commands and opencode can execute them: give it a non-privileged account and a test machine.
Needs our qwen38_exl3_vision plugin and the OrcaSAQ2 plugin; it will not load with stock vLLM, Transformers or SGLang.
Agent tasks: 11–14/30 depending on revision and run (the BF16 original: 16/30), clearly below the 27B models we publish; use it where memory is the constraint.
The calibration of main is tuned to our use (Spanish sysadmin agent with opencode). The public trace improved as much as ours, but other domains were not tested separately.
5.37bpw: capacity per GPU size, speed and the needle test were not measured for it; only KL divergence, the battery above and the loop test.
The 8 / 12 / 16 GB rows were measured on 48 GB GPUs with the KV cache and VRAM sized for each size, not on physical 8 / 12 / 16 GB cards.
Vision: only single images (≤1 MP) were tested; not several images, video or photographs.
Quality was not evaluated with standard benchmarks. All numbers come from one software stack.

Feedback welcome

Credits & license

Base model: Jackrong/Kestrel-4B-Coder (Apache-2.0), a fine-tune of Qwen3.5-4B.
exllamav3 and the EXL3 format by turboderp (MIT), including qbench and the public evaluation trace; EXL3 is based on QTIP (Tseng et al.).
vLLM integration: the OrcaSAQ2 plugin by Continuum AI Corp; vLLM.
Our code: plugin/qwen38_exl3_vision/, patches/, setup.sh (Apache-2.0).

Kestrel-4B-Coder-EXL3-6.00bpw-8-16GB

README

What is in this repository

Which one to use

Variants / revisions

Files and reproducibility

Results

Quantization quality

Capacity per GPU size

Tool calls and agent tasks of the current main (2026-10-27)

5.37bpw: per-layer bit allocation (2026-10-27)

Test battery

Speculative decoding

Sampling and repetition loops

Known model behaviour and discarded options

Requirements & install

Quick start

Using it with opencode (how we tested)

Important notes

Quantization recipe

Integrity and validation

Limitations

Feedback welcome

Credits & license

Explore FriendliAI today

README

What is in this repository

Which one to use

Variants / revisions

Files and reproducibility

Results

Quantization quality

Capacity per GPU size

Tool calls and agent tasks of the current main (2026-10-27)

5.37bpw: per-layer bit allocation (2026-10-27)

Test battery

Speculative decoding

Sampling and repetition loops

Known model behaviour and discarded options

Requirements & install

Quick start

Using it with opencode (how we tested)

Important notes

Quantization recipe

Integrity and validation

Limitations

Feedback welcome

Credits & license

Tool calls and agent tasks of the current `main` (2026-10-27)

`5.37bpw`: per-layer bit allocation (2026-10-27)

Tool calls and agent tasks of the current `main` (2026-10-27)

`5.37bpw`: per-layer bit allocation (2026-10-27)