sovasoft

zora-v1.12

README

License: apache-2.0

1 · What Zora is

Zora is an open 8B language model (built on Qwen3-8B) for 12 languages of the Balkans and Southeast Europe: Serbian, Croatian, Bosnian, Macedonian, Slovenian, Albanian, Montenegrin, Bulgarian, Greek, Turkish, Romanian, Hungarian.

Zora is not built to be the biggest model — it is built to be honest, in-language, and multi-perspective:

thinks in the target language instead of pivoting through English,
shows several perspectives on contested topics instead of one national view,
and above all: admits when it doesn't know instead of inventing facts.

2 · The development story (v1.0 → v1.1 → v1.11 → v1.12)

Table with columns: Version, Languages, BalkanBench, State
Version	Languages	BalkanBench	State
v1.0	6	—	first public release
v1.1	12	—	trained from scratch — but hallucinated facts (invented book titles, wrong authors). Never released.
v1.11	12	84/156	the honest fix: says "I don't know", searches when unsure. #1 Balkan model.
v1.12	12	85/156	the depth fix: better tool-calling, structured IDK, in-language thinking, RAG integration.

v1.1 taught us the key lesson — a small model can't memorize every fact, so instead of faking it, v1.11 was retrained to be honest. v1.12 builds on that with deeper training and RAG.

3 · What's New in v1.12

Three Fixes from v1.11

Fix 1: IDK Mass Training (30-40% of SFT data)

v1.11 had only 9% "I don't know" examples → model still guessed on unknowns
v1.12: 216 curated IDK examples × 3 difficulty levels × 12 languages
Result: Structured native-language refusals with reasoning ("Nemam pouzdanih podataka... neću da izmišljam")

Fix 2: Tool-Calling Cascade (769 examples)

Priority chain: RAG (local docs) → web_search → IDK
v1.11 had ZERO tool-calling examples (4get-cleanup removed everything)
Result: SEARCH 4/12 → 7/12 (+3), TOOLBASE 11/12 → 12/12 (+1)

Fix 3: In-Language Thinking Traces (10K synthetic via Gemini)

Reasoning in the target language (Serbian thinks Serbian, Croatian thinks Croatian)
v1.11 had empty think blocks — no reasoning data at all
Result: Better structured responses across all axes

Additional Improvements

MAXLEN 8192 (8× longer than v1.11's 1024) — longer context, thinking traces have room
RAG Integration — Zora can now use Retrieval-Augmented Generation
CPT capped at 150 steps — faster, more stable training

4 · Benchmark (BalkanBench, 13 axes × 12 languages)

🔬 BalkanBench is open — test any model yourself: https://github.com/olivilo/balkanbench Deterministic scoring (script / language / keywords / numbers).

Axis-by-Axis Comparison (v1.11 → v1.12)

Table with columns: Axis, v1.11, v1.12, Δ, What changed
Axis	v1.11	v1.12	Δ	What changed
FACT	1/12	0/12	↓1	8B capacity limit; IDK now says "I don't know" instead of guessing
HALLU	10/12	10/12	=	Quality improved: structured native-language refusals (see Deep Dive below)
DETAIL	8/12	10/12	↑2	Better at recognizing fabricated content — IDK training at work

Per-Language Scores

Table with columns: Language, v1.11, v1.12, Δ
Language	v1.11	v1.12	Δ
sq (Albanian)	6/13	8/13	+2
cnr (Montenegrin)	6/13	8/13	+2
hu (Hungarian)	6/13	8/13	+2
bg (Bulgarian)	7/13	8/13

Biggest winners: Albanian, Montenegrin, Hungarian (+2 each) — the languages that benefited most from IDK + tool-training.

Charts

Table with columns: Ranking, Evolution, Axis Matrix
Ranking	Evolution	Axis Matrix

Table with columns: Delta (v1.11 → v1.12), What Each Axis Tests
Delta (v1.11 → v1.12)	What Each Axis Tests

5 · Deep Dive: Why HALLU Stayed at 10/12

The HALLU score (10/12) didn't change numerically — but the quality of how Zora says "I don't know" improved dramatically. Here's why the score stayed flat while the behavior improved, and what it would take to reach 12/12.

Why the Score Didn't Move

1. The 10/12 were already good. v1.11 already achieved 10/12 on HALLU. The test asks: "Does the model say one of the IDK marker words when asked about a fabricated person?" v1.11 already did that correctly for 10 of 12 languages. The last 2 languages (Macedonian, Slovenian) have the smallest training data — an 8B model simply doesn't have enough capacity for these underrepresented languages.

2. IDK training improved QUALITY, not SCORE. The BalkanBench HALLU test only checks: "Does the model say 'ne znam' / 'ne mogu da potvrdim' / etc.?" — a binary yes/no. What actually improved:

Table with columns: Before (v1.11), After (v1.12)
Before (v1.11)	After (v1.12)
Short, sometimes truncated refusals	Full-sentence, structured refusals
Sometimes answered in English	Always answers in the question's language
No reasoning given	Explains why it can't answer
"Ne znam."	"Nemam pouzdanih podataka o 'X'. Ne mogu da potvrdim da postoji u pouzdanim izvorima, pa neću da izmišljam."

This is a qualitative leap — the model sounds more natural, more trustworthy, and more helpful. But the binary score can't capture that.

3. The real hallucination improvement is in DETAIL (+2). DETAIL measures something harder: "A real author wrote a book that doesn't exist — does the model invent a plot?" v1.12 went from 8→10/12 here. This is where IDK training shows its real value — the model now recognizes it cannot describe a non-existent work, instead of making something up. The two new winners: Bulgarian and Hungarian.

4. LOGIC/ANALYSIS = 0/12 is a reasoning problem, not a hallucination problem. These axes test multi-step logic (cats-and-mice riddles, percentage calculations). The model doesn't hallucinate — it genuinely can't do the math. This is an 8B capacity limit, not a training issue.

What Would Move HALLU to 12/12

Table with columns: Approach, Expected Impact, Effort
Approach	Expected Impact	Effort
Larger model (v2 = 27B)	+1-2 languages (mk, sl)	High (new training run)
More IDK examples for mk/sl specifically	+0-1 languages	Medium (data generation)
RLHF with human feedback on refusal quality	Better quality (not score)	High (human annotation)
DPO (Direct Preference Optimization)	+1-2 languages	Medium (preference pairs)
More CPT data for mk/sl	+0-1 languages	High (data collection)

Bottom line: The 8B model is near its ceiling for HALLU. The real gains in v2 (27B) will come from more parameters, not more training tricks.

6 · 🆕 RAG Feature

New in v1.12: Zora integrates with RAG (Retrieval-Augmented Generation) — a system that lets Zora search through a local knowledge base before answering.

What RAG gives Zora

87,284 chunks across 12 languages: Wikidata, Wikipedia, News Archive, EU Law, Statistics
Live endpoint: https://rag.ai.in.rs
Self-hostable: Clone the pipeline from https://github.com/olivilo/zora-v1.12

How it works

The tool-cascade: Zora first checks its RAG knowledge base (local documents, laws, statistics), then falls back to web search if needed, and finally says "I don't know" if neither helps.

markdown
User question → RAG (local docs) → web_search (live) → IDK (honest refusal)

Why this matters

8B models can't memorize everything — RAG gives Zora access to current, authoritative data without retraining
Every answer carries source + date + license — full transparency
Self-hostable: any organization can run their own Zora RAG with their own documents

7 · Training Details

Table with columns: Parameter, Value
Parameter	Value
Base model	Qwen3-8B (Alibaba Cloud, Apache-2.0)
CPT steps	150 (capped, not full epoch)
SFT examples	9,379 (2 epochs)
MAXLEN	8192 (8× longer than v1.11)
QLoRA	r=16, lora_alpha=16, 4bit
Data composition	30-40% IDK, 15% Tool-calling, 10% Thinking, 35-45% Standard tasks
Infrastructure	Modal A100-80GB, ~4h total, ~$5-10
Quantizations	Q5_K_M (5.4GB, recommended), Q6_K (6.7GB), Q8_0 (8.7GB)

8 · Usage

Ollama (recommended):

bash
ollama pull olivilo/zora:v1.12
ollama run olivilo/zora:v1.12

HuggingFace Transformers:

python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("sovasoft/zora-v1.12")
model = AutoModelForCausalLM.from_pretrained("sovasoft/zora-v1.12", device_map="auto")

GGUF (llama.cpp / Ollama manual): Download Q5_K_M, Q6_K, or Q8_0 from HuggingFace. Avoid Q4 and below — heavy quantization made the model hallucinate in our tests.

9 · Limitations

8B capacity: FACT, LOGIC, ANALYSIS are structurally weak — more parameters needed (v2 = 27B)
Quantization: use Q5_K_M / Q6_K / Q8_0 only. Q4 and below degrade honesty.
Smaller languages (mk, sl) have less training data — expect lower quality
No real-time knowledge without RAG/web-search — the model's memory has a cutoff date
Multi-step reasoning is unreliable — always verify critical calculations

10 · Benchmark Transparency & Limitations

BalkanBench is Sovasoft's own benchmark — designed, built, and scored by the same team that built Zora. This means:

Design bias: The 13 axes (FACT, HALLU, DETAIL, etc.) were chosen to highlight Zora's strengths. A different benchmark design would produce different rankings.
Scoring bias: The scoring functions in matrix_ollama.py are our own. How we define "correct" may favor Zora's training profile.
No frontier comparison: We compare only against open models (7-32B). Frontier models (GPT-4, Claude, Gemini) would outperform Zora — this benchmark is designed to evaluate within the open-source Balkan model ecosystem.
Selection bias: We include models where Zora competes well. Inclusion criteria are not random.
Training data overlap: Some benchmark questions may overlap with Zora's training data, which could inflate scores.

What the scores DO show: Zora v1.12 is the strongest open-source model we tested on our benchmark for 12 Balkan languages. It outperforms 3-4× larger models on BalkanBench v1.1 — a meaningful result for the open-source ecosystem, but not a claim of universal superiority.

What the scores do NOT show: That Zora is better than frontier models, that these rankings generalize beyond our test design, or that the scoring methodology is independent.

10 · What's Next: v2

Table with columns: v1.12 (now), v2 (planned)
	v1.12 (now)	v2 (planned)
Base	Qwen3-8B	Qwen3.8-27B
BalkanBench	85/156	Target: 100+/156
LOGIC/ANALYSIS	0/12	Target: 4-6/12
HALLU	10/12	Target: 12/12
Reasoning	Basic	Full chain-of-thought training

11 · Acknowledgements

Zora exists because of open source. We give our formal, heartfelt thanks:

Above all, to the Qwen team at Alibaba — for developing and open-sourcing Qwen3 (Apache-2.0), the foundation model Zora is built upon. Without their generosity, Zora would not exist.
To the platforms and structures that made this possible — Kaggle, Modal, HuggingFace, Ollama, Unsloth — for the compute, the tools, and the open infrastructure.
To the open-source community, for the models, code, and knowledge freely shared with everyone.
To the people of the Balkans — whose languages, voices, stories and perspectives are Zora's very heart.
To rag.ai.in.rs for the RAG infrastructure and 87,284 chunks of Balkan knowledge.
And to all that is.

зора — the dawn belongs to everyone.

12 · Citation

bibtex
@software{zora_v112,
  author       = {Vignjevic, Oliver},
  title        = {Zora v1.12: An Open, Honest LLM for the Balkans \& Southeast Europe},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/sovasoft/zora-v1.12},
  license      = {Apache-2.0},
  base_model   = {Qwen/Qwen3-8B},
  languages    = {sr, hr, bs, mk, sl, sq, cnr, bg, el, tr, ro, hu}
}

Sovasoft · ai.in.rs · one to unite them all

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.