1. Model Introduction
Smaug-Decisions is a 4B decision model that hears, sees and reads. A fine-tune of Google's
Gemma 4 E4B-it by Abacus.AI, it answers a typed question about audio, an image or
text with a calibrated probability per option, in one forward pass and with no generated text.
It is the only 4B decision model with results on all three input types. On audio it is ahead of the frontier APIs: 60.1 skill on
Earshot, our 6,671-item audio decision benchmark, against 59.4 for Gemini 4 Argon and 50.5 for Gemini 3.8 Flash, and better
calibrated than both (ECE 0.069 vs 0.123 and 0.204). On the public Jev Decision Index it scores 57.9 on images and 56.6 on text,
the best Gemma 4 E4B on both.
import sysfrom huggingface_hub import snapshot_downloadsys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))from smaug_decide import SmaugDecide # ships with this repo model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions")model("call.wav", "Has the speaker finished their turn?") # -> {'Yes': 0.93, 'No': 0.07}
2. Highlights
- Ahead of the frontier APIs on audio, and better calibrated, so the probabilities work as thresholds.
- Hears, sees and reads: audio up to 30 s, images and text through one prompt and one readout.
- One pass, no decoding: about 200 ms per audio decision on an H100.
- Runs on phones: Smaug-Decisions-LiteRT is 3.45 GB, keeps 59.4 audio skill and answers
a text question in about 0.6 s on an iPhone 14 Pro Max, fully offline.
3. Model Summary
4. Evaluation
Audio: Earshot
Earshot is our audio decision benchmark: 6,671 items and 17.2 hours of audio in four tracks: conversation dynamics (turn
completion, next speaker, speaker count, speech activity; from VoxConverse, AMI and Smart Turn), external turn completion (Smart
Turn v3.2 test), sound and music (genre, instrument, tempo, scene, sound events) and a small pilot track. Audio skill is
the mean task skill per track, averaged over the tracks: 0 is guessing, 100 is perfect. No benchmark recording or item is in the
training data.
Need it faster? Smaug-Audio-Decisions-0.7B scores 53.0 on Earshot at 9.7 ms; in front of this
model as a cascade it matches this model's accuracy (60.1) at 130 ms
instead of 210 ms.
Images and text: the public Jev Decision Index
The matrix at the top has the headline numbers: 57.9 on images and 56.6 on text, the best Gemma 4 E4B on both (Winnow-E4B, the
other Gemma 4 E4B fine-tune on the boards: 48.1 and 41.7). Images: our run of the official vision kit (0.4, public sets) at 560 image
tokens.
5. Training Approach
- Readout, not generation. The answer is read from the LM-head logits of the option letters at the answer position (tanh
softcap 30, then softmax; select-all uses a sigmoid with one learned threshold). A decision costs one prefill.
- Decision-tuned and calibrated. Full fine-tune of Gemma 4 E4B-it on a broad mix of typed decision tasks over audio, images and
text, with calibration as an explicit objective and options shuffled so the model learns the options, not their positions.
6. Limitations
- Decisions, not text: each call answers one typed question with a probability per option; the model does not write text.
- Per request: one audio clip of up to 30 s (longer audio keeps its last 30 s), one image at 560 image tokens, up to 20 options.
- Thresholds: set decision thresholds on your own data for high-stakes uses.
7. Usage
pip install "transformers>=5.20" torch soundfile pillow huggingface_hub
import sysfrom huggingface_hub import snapshot_downloadsys.path.insert(0, snapshot_download("abacusai/Smaug-Decisions", allow_patterns=["smaug_decide.py"]))from smaug_decide import SmaugDecide model = SmaugDecide.from_pretrained("abacusai/Smaug-Decisions") # CUDA, Apple Silicon (MPS) or CPU, picked automaticallymodel("call.wav", "Has the speaker finished their turn?") # predicate -> {'Yes': p, 'No': 1-p}model("song.wav", "Which genre is playing?", ["rock", "jazz", "classical"]) # choice -> probabilities sum to 1model("room.wav", "Which sounds are present?", ["speech", "music", "alarm"], multi=True) # select-all -> one probability eachmodel("receipt.jpg", "Does the total exceed $50?") # imagesmodel("The order never arrived.", "How upset is the customer?", ["calm", "annoyed", "angry"], descriptions=["neutral tone", "complains", "threatens to leave"], rules="Judge the tone, not the facts.")
from_pretrained downloads the checkpoint (about 16 GB). Any runtime can reproduce the readout: run the prompt in
smaug_decide.render, take the LM-head logits of the tokens A..T at the last position, apply the softcap from readout.json,
then softmax (one answer) or sigmoid(z - b) with b from readout.json (select-all). With plain Transformers,
AutoModelForMultimodalLM.from_pretrained("abacusai/Smaug-Decisions", dtype="bfloat16") loads the weights.
On phones
Smaug-Decisions-LiteRT is this model in one 3.45 GB file for Google's on-device
runtime LiteRT-LM, running on the phone GPU. At about a fifth of the size it keeps almost all of this model's quality. It runs in
the Cascade app on iPhone and Android next to Smaug-Audio-Decisions-0.7B, which answers the easy
audio questions first. App source: github.com/abacusai/Smaug-Decisions-Mobile.
Table with columns: Smaug-Decisions (16 GB), Smaug-Decisions-LiteRT (3.45 GB), change | Smaug-Decisions (16 GB) | Smaug-Decisions-LiteRT (3.45 GB) | change |
|---|
| Earshot audio skill | 60.1 | 59.4 | −0.7 |
| Earshot calibration error (ECE, lower is better) | 0.069 | 0.074 | +0.005 |
| Decision Index text | 56.6 | 53.9 | −2.7 |
| Decision Index images, 560 image tokens* | 57.2 | 55.1 | −2.1 |
| Decision Index images, 280 image tokens* | 54.8 | 51.8 | −3.0 |
Same items, same harness; the phone-build scores come from a PyTorch replica of the file's quantized weights. *10 of the 11 public
vision sets (Winoground left out; this model's 11-set index is 57.9).
Table with columns: iPhone 14 Pro Max (6 GB RAM, phone GPU), measured| iPhone 14 Pro Max (6 GB RAM, phone GPU) | measured |
|---|
| first load (one time) / later loads | 13-21 s / 0.48 s |
| text decision | ~0.6 s |
| audio decision | ~1.6 s |
| image decision (280 image tokens) | ~7.4 s |
| peak app memory, with Smaug-Audio-Decisions-0.7B loaded | 1.35 GB |
| storage | the 3.45 GB file + a ~2.4 GB cache written on first load |
8. License
Apache 2.0, the same license as the base model, Gemma 4 E4B-it.
9. Citation
@misc{abacusai2026smaugdecisions, title = {Smaug-Decisions: calibrated typed decisions over audio, images and text}, author = {Abacus.AI}, year = {2026}, url = {https://huggingface.co/abacusai/Smaug-Decisions}}
If you have any questions, please reach out at Abacus.AI.