Highlights
- Release-set mean 77.5, up from 75.4 (v0.2) and 75.1 (Qwen3.5-4B, its original base).
- Against v0.2: 4 real wins, 0 real losses. GPQA Diamond +11.1, IFEval +6.7, MATH-500 +4.0, GSM8K +3.8.
- Against Qwen3.5-4B: 4 real wins (GPQA Diamond +13.6, BFCL tool calling +4.7, MATH-500 +3.2, Belebele Serbian +3.2),
1 real loss (TruthfulQA −2.8).
- Ties Qwen3.5-9B on GSM8K, MATH-500, MMLU-Pro, GPQA Diamond, IFEval and BFCL (paired, same questions).
The 9B is still ahead on knowledge (MMLU, TruthfulQA, HellaSwag, Serbian knowledge) and HumanEval+.
- Thinking mode: GPQA Diamond 71.2% (v0.2: 62.6%), MMLU-Pro 80.5%, GSM8K 87.0%.
- Shorter, decisive answers: on GPQA it ran out of room on 21% of questions with thinking off (v0.2: 49%).
- Still knows who it is (100% with no system prompt) and keeps customer personas (100%).
How it was made
- v0.2: Qwen3.5-4B → Serbian continued pretraining → fine-tune (~31,000 examples) → 80/20 blend with Qwen3.5-4B.
- Prompts: 8,000 tasks with answers a program can check: instruction-following constraints (word counts, formats,
casing, forbidden characters), maths with a known final answer, and coding tasks with unit tests. Decontaminated
against the evaluation sets (13-gram overlap with IFEval, GSM8K, MMLU, HumanEval, MBPP, MATH-500, AIME).
- Filter: v0.2 answered each prompt 4 times; we kept the 817 prompts where its answers got different scores.
Prompts it always passes or always fails teach nothing.
- GRPO: 600 steps, 2 prompts × 8 answers per step, LoRA rank 32, learning rate 2e-5, thinking off, answers up to
640 tokens. Each answer is scored by the checker (rule check, exact answer, or tests passed); answers cut off by
the length limit score nothing. About 6 hours on one RTX 4060 Ti 8 GB. The LoRA is merged into the full weights.
Benchmarks (thinking off)
Same harness, prompts, settings and question samples for every model; Q4_K_M GGUF on llama.cpp, no system prompt.
✅ / ❌ = a real win / loss against v0.2 (beyond 2 standard errors, paired on the same questions); unmarked rows are ties.
Best 4B per row in bold.
Table with columns: Benchmark, Quata2 4B v0.3, Quata2 4B v0.2, Qwen3.5 4B, Qwen3 4B 2507, Gemma 4 E4B, Qwen3.5 9B| Benchmark | Quata2 4B v0.3 | Quata2 4B v0.2 | Qwen3.5 4B | Qwen3 4B 2507 | Gemma 4 E4B | Qwen3.5 9B |
|---|
| MMLU | 71.8 | 73.0 | 70.6 | 64.6 | 65.0 | 80.0 |
| MMLU-Pro | 71.3 | 68.7 | 71.3 | 66.3 | 66.0 | 72.3 |
| GPQA Diamond ✅ | 51.0 | 39.9 | 37.4 | 29.3 | 40.4 | 49.5 |
| ARC-Challenge | 92.6 | 91.6 | 92.4 | 92.0 | 89.2 | 95.0 |
| HellaSwag | 86.4 | 86.0 | 86.0 | 84.6 | 80.6 | 92.4 |
| TruthfulQA (MC1) | 70.2 | 71.0 | 73.0 | 67.0 | 66.2 | 77.6 |
| GSM8K ✅ | 89.4 | 85.6 | 86.6 | 88.8 | 86.4 | 89.6 |
| MATH-500 ✅ | 83.4 | 79.4 | 80.2 | 77.2 | 77.2 | 83.6 |
| IFEval (strict) ✅ | 85.5 | 78.8 | 82.3 | 83.6 | 84.7 | 83.8 |
| BFCL tool calling | 89.9 | 88.7 | 85.2 | 91.7 | 89.9 | 88.6 |
| HumanEval+ | 79.3 | 77.4 | 75.6 | 84.8 | 85.4 | 86.6 |
| MBPP+ | 67.7 | 66.9 | 64.8 | 63.0 | 67.5 | 63.5 |
| Belebele Serbian | 87.8 | 86.6 | 84.6 | 84.2 | 86.8 | 90.2 |
| Global-MMLU Serbian | 58.6 | 60.0 | 58.4 | 50.0 | 51.8 | 67.0 |
| Serbian LLM Eval (generative) | 77.4 | 76.8 | 78.2 | 73.9 | 75.9 | 82.4 |
| Mean | 77.5 | 75.4 | 75.1 | — | — | 80.1 |
Thinking mode
Thinking on, up to 32k tokens, sampling 0.6 / 0.95 / 20, same 200 / 198 questions for both.
Table with columns: Benchmark, v0.3, v0.2, Finished within 32k (v0.3 / v0.2)| Benchmark | v0.3 | v0.2 | Finished within 32k (v0.3 / v0.2) |
|---|
| GSM8K | 87.0 | 85.0 | 100% / 100% |
| MMLU-Pro | 80.5 | 81.0 | 100% / 98% |
| GPQA Diamond ✅ | 71.2 | 62.6 | 98% / 88% |
MATH-500, SuperGPQA, BIG-Bench Hard and AIME in thinking mode have not been rerun for v0.3 yet
(v0.2: 95.0 / 47.0 / 86.7 / 51.7).
Quata's own checks
Targeted tests with the website's system prompt. Small sets, so read them as directional.
Table with columns: Check, v0.3, v0.2, Qwen3.5 4B, Items| Check | v0.3 | v0.2 | Qwen3.5 4B | Items |
|---|
| Knows it is Quata2 (no system prompt) | 100.0 | 100.0 | 0.0 | 36 |
| Never mentions itself unprompted | 100.0 | 100.0 | 98.3 | 300 |
| Stays in a customer persona | 100.0 | 100.0 | 97.0 | 100 |
| Rejects a false premise | 100.0 | 75.0 | 100.0 | 4 |
| Holds a correct answer under pushback | 90.0 | 75.0 | 65.0 | 20 |
| Fixes a real mistake when challenged | 87.5 | 80.0 | 95.0 | 40 |
| Knows where companies are | 91.7 | 91.7 | 91.7 | 12 |
| Facts about Serbia | 50.0 | 75.0 | 62.5 | 8 |
| Real tool-using agent tasks, solved | 81.5 | 85.4 | 80.1 | 151 |
| Multi-step agent tasks, solved | 72.5 | 78.4 | 74.5 | 102 |
None of the differences between v0.3 and v0.2 here are beyond noise.
Reading the numbers: what is noise
Every benchmark is a sample of questions, so every score has a margin of error. We call a difference real only
when it is larger than twice its standard error, computed pair by pair on the same questions. On 500 questions that
is about 2–4 points; on the small checks above it is 15–25 points. Anything smaller is a tie.
Get started
from transformers import AutoTokenizer, Qwen3_5ForConditionalGeneration model_id = "M1n1A1/MiniAI-Quata2-4b-v0.3"tok = AutoTokenizer.from_pretrained(model_id)model = Qwen3_5ForConditionalGeneration.from_pretrained( model_id, torch_dtype="auto", device_map="auto") messages = [{"role": "user", "content": "Ko te je napravio?"}]prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False) # True = think step by step firstinputs = tok(prompt, return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=512)print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
The safetensors include the Qwen3.5 vision tower (unchanged from v0.2); the GGUF is text-only.
Thinking mode: enable_thinking=True, or put /think / /no_think in a message. Give it room when it thinks:
our thinking-mode numbers use up to 32,000 new tokens.
Run it with llama.cpp
llama-server -m MiniAI-Quata2-4b-v0.3-Q4_K_M.gguf --jinja -c 8192
Recommended sampling: temperature 0.7, top_p 0.8, top_k 20 (thinking: 0.6, 0.95, 20).
Details
- Architecture: Qwen3.5-4B, 4.2B language parameters (+0.33B vision, +0.12B MTP), 32 layers, hidden size 2560, hybrid attention
- Weights: Quata2 4B v0.2 + merged GRPO LoRA (rank 32); vision tower, MTP head and embeddings unchanged
- Context length: 262,144 tokens in the model config; RL used answers up to 640 tokens
- Precision: bf16 safetensors (~9.7 GB with MTP); Q4_K_M GGUF (~2.8 GB)
- Languages: English first; Serbian, Croatian and Bosnian improved, grammar still behind English
- License: Apache 2.0 (same as the base model)
Honest caveats
- RL improves skills, not knowledge: knowledge benchmarks (MMLU, Global-MMLU Serbian, TruthfulQA) are unchanged or
slightly lower, and TruthfulQA is a real loss against Qwen3.5-4B.
- Agent tasks and Serbia facts are a little lower than v0.2 (within noise); in one rerun it named a weekday without
checking it with its date tool. Check answers that depend on tools.
- Qwen3 4B 2507 and Gemma 4 E4B are still ahead on HumanEval+; Qwen3 4B 2507 on BFCL.
- The wider public suite from the v0.2 card (MMLU-Redux, SuperGPQA, Winogrande, BBH, MuSR, DROP, Belebele Croatian,
MMLU-ProX Serbian, YugoGPT's log-likelihood suite) has not been rerun for v0.3 yet.
Citation
@misc{miniai2026quata2v03, title = {MiniAI Quata2 4B v0.3}, author = {{MiniAI}}, year = {2026}, publisher = {Hugging Face}, howpublished = {\url{https://huggingface.co/M1n1A1/MiniAI-Quata2-4b-v0.3}}, note = {Quata2 4B v0.2 trained further with GRPO on verifiable rewards}}