surogate

rune-26b-a4b-GGUF

README

License: apache-2.0

Accuracy

Decision Index is a public leaderboard of decision models. Its current edition, 0.2.1, scores the 0.2 suite (151,034 requests from 44 benchmarks) on a 38-benchmark panel: each benchmark's accuracy is rescaled so that chance scores 0 ("skill"), averaged within five areas, and the areas are weighted into the headline index. Rune v3's row is the leaderboard's: our submitted run of these bf16 weights, adopted after the maintainers' own reproduction matched it bit for bit. Index and area skill scores in %, as published on 26 September 2026:

Table with columns: System, Index, Raw, Knowledge & Reasoning, Language, Retrieval & Classification, Tools & Automation, Arts & Human Taste
System	Index	Raw	Knowledge & Reasoning	Language	Retrieval & Classification	Tools & Automation	Arts & Human Taste
Surogate Rune v3, `thinking: true` (bf16)	59.2*		48.0	64.0	64.7	72.4	41.3
Jev 1.13	57.89	68.08	51.3	62.0	55.4	75.1	37.7
Surogate Rune v3 (bf16)	57.44	67.30	43.4	63.1	63.5	71.2	41.9
AutoJev-27B	56.40	66.89	40.9	63.5	54.9	79.3	39.4
simple-jev (Qwen3.8-27B)	55.74	66.26	36.6	62.1	63.2	76.2	36.5
Decider chat (Qwen3.6-27B)	51.35	63.06	37.0	57.1	52.2	71.4	35.1
Winnow-12B (Q8_0)	50.02	61.91	33.8	56.0	54.0	71.0	30.0
JoshuaSP diffusiongemma (26B-A4B)	49.47	61.28	32.7	53.5	58.4	70.2	26.7
Jevfire	49.37	61.63	30.5	53.3	56.2	72.3	32.2
Decider 35B-A3B (NVFP4)	47.11	59.72	31.8	55.5	54.7	56.5	32.6

* Our estimate, not yet on the leaderboard: the leaderboard's per-benchmark 0.2.1 scores for Rune v3, each moved by the change thinking made to that benchmark in our full run of the suite with the official 0.2 scorer (the 0.2.1 scorer is not public yet). Rebuilt the same way, the published rows for Rune v3 (57.44) and Jev 1.13 (57.89) come out exactly.

The index counts unanswered requests as wrong. Rune v3 is second on the board, 0.45 points behind Jev 1.13, and with thinking on it would lead it by about 1.3. On the previous edition, 0.2 (40 benchmarks, a plain mean of the areas), Rune v3 scored 53.39, and 54.89 with thinking, against 51.67 for Jev 1.13. 0.2.1 takes RouterBench and SGD out of the panel, weighs Knowledge & Reasoning and Language at 26% each and Arts & Human Taste at 10%, and rescores five benchmarks.

Calibration

Measured on the Decision Index's 33 benchmarks that have a right or wrong answer per field (each benchmark weighted equally, every request). At the default temperature Rune is overconfident; reading the answers at temperature 2 makes the probabilities match accuracy:

Table with columns: decision temperature, accuracy, mean confidence, ECE, Brier
decision temperature	accuracy	mean confidence	ECE	Brier
1 (default)	73.9%	86.4%	12.5%	0.385
2	73.9%	73.6%	2.2%	0.360

The temperature changes only the probabilities, never the chosen option. With surogate, pass --decision-temperature 2 to surogate serve.

Thinking

Thinking is opt-in, per request. With "thinking": true, a question Rune is unsure of thinks before it answers: if its one-pass confidence (the top option's probability; for a true/false question the larger of p and 1 - p) is below 0.7, Rune reasons for up to 512 tokens and then answers; every other question keeps its one-pass answer.

On the Decision Index 0.2 suite (our full run, scored with the official 0.2 scorer), about one question in ten thinks and the 0.2 index rises from 53.39 to 54.89, with Knowledge & Reasoning from 43.5 to 48.4; under 0.2.1's weights that is about 59.2 against 57.44 (see Accuracy). The largest gains in skill points: GSM8K +15.2, CRUXEval +11.4, CLadder +10.4, BBH +8.0, PhishNChips +5.1. ForecastBench, whose answers are probabilities, loses 9.8: a thought makes those probabilities more extreme.

A question that thinks takes about 5 seconds instead of 0.2 (see Speed); the others are unchanged. Thinking is not available together with images yet.

Images

The vision tower is kept, so Rune accepts images. With surogate, start the server with --vision and put the picture in the request's images array as a data URL; it is placed ahead of the text.

We measured image decisions with the item format and scoring of Image JevBench v0.1 on 1,196 items: the benchmark's 136 public items (its 8 example cards, and its 128 preview items rebuilt from their original sources) and 1,060 items we built the same way from the same openly licensed sources. This is our own run, not a leaderboard result: the benchmark's sealed items are private and were not used, so our items are a different sample. Accuracy in %, one request at a time on one RTX PRO 6000 with surogate 1.5.3, every item answered:

Table with columns: Family, Items, Chance, Rune v3, 1,120 image tokens, Rune v3, 280 image tokens (default), Gemma 4 26B-A4B-it, 280 image tokens
Family	Items	Chance	Rune v3, 1,120 image tokens	Rune v3, 280 image tokens (default)	Gemma 4 26B-A4B-it, 280 image tokens
All items	1,196	29.6	83.4	77.1	71.8
The benchmark's public items	136	27.5	79.4	75.7	73.5
ScreenSpot (click target on a screenshot)

The screen families ask which of five labelled markers drawn on the screenshot to click. The benchmark's own everyday photos are synthetic and mostly sealed, so that row is 100 real VizWiz photos plus the benchmark's 2 example photos; ArxivQA's answer labels were machine-generated.

Image probabilities are well calibrated at the default decision temperature (ECE, ten bins, top option):

Table with columns: accuracy, mean confidence, ECE at temperature 1, ECE at temperature 2
	accuracy	mean confidence	ECE at temperature 1	ECE at temperature 2
Rune v3, 1,120 image tokens	83.4%	88.2%	5.1%	6.5%
Rune v3, 280 image tokens	77.1%	84.0%	6.9%	5.6%
Gemma 4 26B-A4B-it, 280 image tokens	71.8%	94.7%	22.9%	16.8%

The --decision-temperature 2 recommended above for text keeps image ECE within 1.5 points of temperature 1.

Image token budget. Gemma 4 turns each image into at most 280 tokens by default. Raising the image processor's budget to 1,120 (max_soft_tokens and image_seq_length in processor_config.json) gives Rune more detail: +6.3 points over all items (paired 95% interval +4.3 to +8.4), most of it on small screen targets (ScreenSpot-Pro +20.8) and dense pages and tables (Mind2Web +14.7, FinQA +10.7). It costs time: one image decision took 0.18 s at 280 tokens and 0.46 s at 1,120 (median, one request at a time, RTX PRO 6000).

Speed

Measured during the Decision Index run above: one NVIDIA RTX PRO 6000 Blackwell per server, bf16, surogate 1.5.3 with the MoE routing fix described below, four concurrent clients per GPU. A Decision Index request carries a shared state and often many questions, so these figures are per request, not per question:

Table with columns: median, p95, mean
	median	p95	mean
latency per request	388 ms	2.33 s	751 ms

One question per request, one request at a time, on an idle RTX PRO 6000 (surogate main):

Table with columns: median, p90
	median	p90
a decision (no thinking)	0.09-0.18 s	0.2 s
a question that thinks (`thinking: true`, confidence below 0.7)	4.9 s	6.3 s

With thinking on, questions at or above the confidence gate answer as fast as without it. On the Decision Index's mix, where one question in ten thinks, that averages about 0.65 s per question.

Running it

Thinking is off, and that is the default: the chat template in these files disables it.

With surogate (recommended)

The weights take 51.6 GB, so they need a GPU with room for them plus the KV cache; we serve them on 96 GB RTX PRO 6000 cards:

bash
hf download surogate/rune-26b-a4b-GGUF --local-dir rune-v3
surogate serve ./rune-v3 --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 --max-num-seqs 8 --kv-capacity auto --decision-temperature 2

Under sustained concurrent load, surogate 1.5.3 can occasionally hit a non-finite router value in the MoE layers and stop with an illegal memory access. surogate's main branch includes a fix (#217): with it, an affected request fails cleanly and the server keeps running. Use a build that includes it.

Then ask it a decision (use the model id that GET /v1/models lists; add "thinking": true next to "questions" to let unsure questions think, which needs a surogate build that includes #240):

bash
curl http://localhost:8000/api/alpha/decisions -H 'Content-Type: application/json' -d '{
  "model": "<id from /v1/models>",
  "state": "Customer wrote: the package arrived late and damaged, I want my money back.",
  "questions": {
    "refund": {"type": "noul", "instructions": "Does the customer ask for a refund?",
               "criteria": {"true": "A refund is requested", "false": "No refund is requested"}}
  }
}'

With transformers

The checkpoint loads like any Gemma 4 checkpoint. A decision is read at the first generated position as a softmax over the option letters' logits, with the prompt rendered as surogate's decisions endpoint renders it (docs/inference/decisions.md).

There are no GGUF builds of v3 yet.

Files

Table with columns: file, size, contents
file	size	contents
`model-00001-of-00011.safetensors` … `model-00011-of-00011.safetensors`	51.6 GB in total	bf16 weights, at most 5 GB per shard
`model.safetensors.index.json`		tensor-to-shard map
`config.json`, `generation_config.json`		`Gemma4ForConditionalGeneration` config

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

surogate

Model Tree

Base

google/gemma-4-26B-A4B-it

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer