Results
Served with the public basal engine 2.8.0 from the released files on one NVIDIA H100: --mode eager, bf16, one option order (the default), the shipped CALIBRATION.json. Werdykt was read with basal 2.5 (see below).
Table with columns: benchmark, basal-omni-2.0-mini (preview)| benchmark | basal-omni-2.0-mini (preview) |
|---|
| Werdykt v1, macro (hidden PL + EN, 10 categories × 2 languages, 5,000 items) | 0.811 (95% interval 0.801–0.821) |
| Werdykt v1, Polish / English | 0.807 / 0.814 |
| PL decisions (held-out test, 7,081 items) | 0.951 |
| PL general (knowledge, exams, reading; 3,079 items) | 0.767 |
| PL-Real (300 real Polish documents) | 0.960 |
| EN external panel (10 public English decision tasks, 3,842 items) | 0.811 |
| EN corpus test (1,479 items) | 0.692 |
| Coverage at 1% error (share decided automatically with the shipped threshold) | 58.3% at 0.58% observed error |
| Coverage at 5% error | 87.8% at 4.0% observed error |
- Werdykt v1: measured with basal 2.5 (build d72e614),
--mode fast, one option order, on one NVIDIA RTX PRO 6000
Blackwell (bf16). The public engine 2.8.0 gives the same probabilities on our parity set (largest difference 5e-6).
Weakest categories: rules (0.382; compute exact thresholds and dates in code), long documents (0.694) and contracts
(0.726). Strongest: retrieval (0.982), real documents (0.978), routing (0.966).
- Coverage: on the held-out PL decisions and EN corpus tests (8,560 decisions), with the per-type thresholds of the shipped
CALIBRATION.json as the server applies them. At the 1% target, score questions have no certified threshold and go to a person; per type: choice 78.2% at 0.74%, yes/no 43.8% at 0.15%. At 5%: choice 94.7%, yes/no 88.0%, score 47.6%. The test figures stored inside CALIBRATION.json come from the calibration fit (an earlier engine build) and differ slightly; the numbers here are what 2.8.0 serves.
English external panel by task
The EN external panel row above is the mean of these ten public English decision tasks (same run as Results).
Table with columns: task, items, accuracy| task | items | accuracy |
|---|
| FinancialPhraseBank | 300 | 0.957 |
| Belebele (English) | 300 | 0.940 |
| WinoGrande | 300 | 0.907 |
| Circa | 300 | 0.837 |
| TabFact | 300 | 0.830 |
| BIG-Bench Hard | 540 | 0.811 |
| RAGTruth | 300 | 0.780 |
| ContractNLI | 300 | 0.777 |
| TruthfulQA (binary) | 902 | 0.649 |
| JudgeBench | 300 | 0.627 |
| mean of the ten tasks | 3,842 | 0.811 |
Werdykt against other 4B typed-decision models
The same 5,000 hidden Werdykt v1 items (Polish and English), the same client and the same scoring, over each model's
typed-decision API. Each model is served by its maker's own server. The lead is basal-omni-2.0-mini's paired
difference over the same items, in points, with a 95% bootstrap interval.
Table with columns: model, Werdykt v1 [95% interval], Polish / English, basal-omni-2.0-mini lead| model | Werdykt v1 [95% interval] | Polish / English | basal-omni-2.0-mini lead |
|---|
| basal-omni-2.0-mini (preview) | 0.811 [0.801–0.821] | 0.807 / 0.814 | – |
| torchcast-decision-4b (v1.3) | 0.799 [0.788–0.809] | 0.787 / 0.810 | +1.2 [+0.1, +2.2] |
| KnowLine-4B-Gen5.5¹ | 0.776 [0.766–0.787] | 0.764 / 0.788 | +3.4 [+2.3, +4.5] |
| h2o-lightning-4b (v1.1) | 0.753 [0.742–0.764] | 0.745 / 0.761 | +5.8 [+4.7, +6.9] |
| Bielik-One-4B-r1 | 0.696 [0.684–0.709] | 0.687 / 0.705 | +11.5 [+10.2, +12.6] |
- Read between 8 Oct and 13 Nov 2026; torchcast-decision-4b, KnowLine-4B and Bielik-One-4B on one NVIDIA RTX PRO 6000
Blackwell, h2o-lightning-4b on one H100, basal-omni-2.0-mini as in Results.
- ¹ KnowLine-4B was served in FP8; the other models in bf16.
- Werdykt is hidden, so only these totals are shown.
Accuracy by document length
Accuracy on Werdykt (5,000 items, Polish and English; a hidden set, so only these totals are shown) by the length of the
state, in characters (a token is about 2 characters; 1,000 characters is about half a page). 81.1% overall. 95% Wilson
intervals in brackets.
Table with columns: state length, items, accuracy| state length | items | accuracy |
|---|
| 0-1k | 709 | 71.4% [68, 75] |
| 1k-2k | 2,598 | 80.0% [78, 82] |
| 2k-4k | 796 | 92.3% [90, 94] |
| 4k-8k | 394 | 97.5% [95, 99] |
| 8k-16k | 128 | 80.5% [73, 86] |
| 16k+ | 375 | 65.9% [61, 70] |
Short states are mostly rules and abstention items, which are harder for every model, so read the rows as accuracy per
bucket, not as the effect of length alone. The long-document family keeps the task and varies only the length:
Table with columns: state length (long-document family), items, accuracy| state length (long-document family) | items | accuracy |
|---|
| 8k-20k | 226 | 74.3% [68, 80] |
| 20k-30k | 194 | 70.1% [63, 76] |
| 30k+ | 80 | 53.8% [43, 64] |
Same run as the Werdykt row above. The intervals are about the items, not about run-to-run noise.
Quick start
1. Install into a fresh uv environment:
uv venv --python 3.12 --allow-existing ~/basal-env && source ~/basal-env/bin/activateuv pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cu128uv pip install "basal[media] @ https://github.com/rkinas/basal/archive/refs/tags/v2.8.0.tar.gz"
2. Start the server. It prints basal: model ... ready:
basal-serve --model Remek/basal-omni-2.0-mini-preview --name basal-omni-2.0-mini-preview --port 8000
On an NVIDIA GPU the engine picks the mode itself: fast (CUDA graphs) on Hopper with FlashAttention 3 and on
Blackwell workstation GPUs, eager (PyTorch, bf16) on other GPUs. The weights take about 16 GB. A two-way model is served
with --mode eager or --mode fast only; on Apple Silicon pass --mode eager --device mps, because the engine's default there does
not serve two-way models (PyTorch on Metal; several times slower than on an NVIDIA GPU, 32 GB of memory or more recommended). One option order is asked by default; --orders 2 uses CALIBRATION.o2.json automatically.
3. Ask questions:
curl -s localhost:8000/v1/systemone -H 'content-type: application/json' -d '{ "state": "Klient: od wczoraj nie mogę zalogować się do bankowości internetowej, system pokazuje błąd hasła.", "questions": { "dept": {"type": "choice", "instructions": "Do którego działu skierować zgłoszenie?", "criteria": {"cards": "Reklamacje kart", "online": "Wsparcie bankowości elektronicznej", "loans": "Kredyty"}}, "urgent": {"type": "noul", "instructions": "Czy klient nie może korzystać z usługi?"}}}'
Full API (multi, act, tool, option keys, max_error): API.
Speed
One question per request, one option order, on one NVIDIA H100 (bf16), basal 2.8.0:
Table with columns: measurement, --mode fast, --mode eager| measurement | --mode fast | --mode eager |
|---|
| sequential, 200 requests: p50 / p95 per question | 11 ms / 17 ms | 54 ms / 73 ms |
| 32 concurrent clients, 1,000 requests | 123.5 decisions/s | 59.7 decisions/s |
| sequential, states of 4,000-5,000 tokens: p50 per question | 98 ms | 212 ms |
| 16 questions about one state in one request, 8 concurrent | 200 decisions/s | 94 decisions/s |
fast is the default on Hopper GPUs with FlashAttention 3; eager was measured with --media on.
- GPU memory while serving: 20.6 GB (
fast), 31.7 GB (eager with --media on).
--orders 2 costs about twice as much per question.
Files
model.safetensors, tokenizer, chat_template.jinja, processor_config.json: the model (Gemma 4 E4B architecture,
262k vocabulary). Gemma's image and audio encoders are included and unchanged.
CALIBRATION.json: per-type temperatures and certified confidence thresholds for 1% / 5% error, fitted on the served
engine (--mode eager, bf16, one option order).
CALIBRATION.o2.json: the same for --orders 2 (answers read in both option orders and combined).
basal.json: prompt format 2 and the readout protocol.
SHA256SUMS: checksums of the model files.
How it works
The model receives a fixed chat prompt with the state, the question and coded options (A–Z, then two-character codes,
up to 255). The assistant turn is prefilled with {"answer": ", and the decision is the softmax over the next-token
logits of the option codes only: one forward pass, no text generation. The server applies the calibrated temperature of
the question type from CALIBRATION.json.
Use the confidence. CALIBRATION.json stores thresholds for a target error of 1% or 5% among accepted decisions.
They were chosen on calibration data before testing. Accept decisions above the threshold automatically and route the
rest to a person. At the 1% target, score questions have no certified threshold and are deferred.
Limitations
- Preview. This is an early release of the 2.0 mini; the final 2.0 family (nano, mini, max, ultra) comes later.
Validate it on your own documents before relying on it.
- Long prompts: the state is read two-way up to 4,096 prompt tokens (8,192 tokens per request). Longer prompts are
read one way, as in basal-omni-1.0, and the answer does not say which reading was used. Accuracy on the longest
documents is lower (see Accuracy by document length).
- Serving features: for a two-way model the engine turns off the state cache, the prefix cache, reading one state
once for many questions (SOAM) and the long lane. Ask several questions about one state in one request.
- Images and audio: Gemma 4's encoders are included, but this preview was trained and evaluated on text only, and
--media on needs --mode eager. Use it for text; media answers are not evaluated here.
- Exact arithmetic and thresholds are a weak spot of one-pass decisions (Werdykt rules 0.382): compute them in code.
- Option order: one order is asked by default;
--orders 2 reduces sensitivity to option order at about twice the
cost.
- Engine 2.8.0: bf16 only; no FP8 build of this preview.
- GGUF and MLX (experimental, not recommended): basal-omni-2.0-mini-preview-GGUF and basal-omni-2.0-mini-preview-MLX read the state one way only, so they answer differently from this model (their cards give the measured cost). For the model on this card, use basal-serve
--mode fast or --mode eager (on a Mac --mode eager --device mps).
- Review: decisions with serious consequences for people should be reviewed by a person.
Citation
@software{kinas2026basal20, title = {basal-2.0: typed-decision models and inference engine for Polish and English}, author = {Kinas, Remigiusz}, year = {2026}, url = {https://github.com/rkinas/basal}}
License and attribution
Apache-2.0. Fine-tuned from google/gemma-4-E4B-it (Apache-2.0).