namindu94

SinhalaLLM-1.7B

README

License: apache-2.0

Scoreboard

All models were run by us with the same scripts on the same GPU (RTX 5090, bf16). Raw outputs and scripts are in eval/.

Table with columns: Model, Size, Sinhala bits/char (FLORES) ↓, Belebele (Sinhala) ↑, Sri Lankan facts (44 Qs) ↑, Tokens per Sinhala char ↓
Model	Size	Sinhala bits/char (FLORES) ↓	Belebele (Sinhala) ↑	Sri Lankan facts (44 Qs) ↑	Tokens per Sinhala char ↓
SinhalaLLM v2.1 (ours)	1.7B	1.475	34.8%	30% (chat: 43%)	0.244
iCIIT Qwen2.5-1.5B Sinhala	1.5B	2.694	32.2%	0%	1.447
Gemma-2B Sinhala (Udith-Sandaruwan)	2.5B	2.102	30.0%	2%	0.836
Gemma-2-2B Sinhala CoT (anudaw)	2.6B	3.258	31.4%	–	0.836
Qwen2.5-1.5B (general)	1.5B	2.822	31.0%	5%	1.447
SmolLM2-1.7B (our base model)	1.7B	2.889	30.2%	–	2.610
SinLlama, Llama-3-8B (larger reference)	8.1B	1.314	38.0%	75%	0.243

How to read the table:

Sinhala bits/char: how well a model predicts real Sinhala text (FLORES-200 passages). It is fair across different tokenizers; lower is better.
Belebele: 900 four-choice reading-comprehension questions; random guessing scores 25%.
Facts: few-shot, closed-book, scored by whether the answer contains the right answer or an accepted variant.
"–" means not measured.
SinhalaMMLU: not measured yet (dataset access pending).

The honest summary:

Best in its size class. Among models of ≤3B parameters, SinhalaLLM is first on every test:
- Sinhala text prediction: 1.475 vs 2.10 for the next best;
- reading comprehension: 34.8% vs 32.2%;
- Sri Lankan facts: 30% vs 0–5%;
- tokenizer efficiency: 3.4–10.7× fewer tokens.
The 4.7× larger SinLlama 8B is stronger on knowledge and comprehension, as expected for its size. SinhalaLLM:
- uses about 1/5 of the memory: 1.1–1.9 GB as GGUF vs ~5–16 GB for an 8B model;
- generates Sinhala 2.4× faster in the same test (422 vs 178 chars/s);
- runs on an ordinary laptop without a GPU;
- is instruction-tuned for chat, summarisation and RAG out of the box.
Absolute scores are still modest for every small model. Belebele and facts leave plenty of room to improve; see Future work.

Examples

Running locally on a laptop (Q8_0 GGUF, llama.cpp):

Identity Not ChatGPT Summary

Sri Lankan facts it answers correctly:

Longest river Independence Administrative capital

Why SinhalaLLM: compared with other small Sinhala models

These numbers come from the scoreboard above, where every model was run with the same scripts and settings:

Best Sinhala language modelling in its class. 1.475 bits per character on FLORES, vs 2.10 for the best other ≤3B Sinhala model (Gemma-2B Sinhala, which is itself larger). Lower means the model predicts real Sinhala text better.
Best reading comprehension in its class. 34.8% on Belebele (Sinhala), vs 30–32% for the other ≤3B models.
Most Sinhala factual knowledge among small models. 30% (few-shot) and 43% (chat) on a 44-question Sri Lankan facts test, vs 0–5% for the other ≤3B Sinhala and multilingual models.
3.4× fewer tokens than Gemma-based Sinhala models, 6× fewer than Qwen2.5, 10.7× fewer than SmolLM2, thanks to the custom tokenizer below. This is on par with SinLlama's Sinhala-extended tokenizer (0.243), at a fraction of the model size. Fewer tokens means:
- more Sinhala text fits into the context window;
- lower compute per sentence;
- much faster Sinhala generation: 350–420 Sinhala chars/s vs 24–86 for the other small models, and 178 for SinLlama 8B.
Built for use, not just text completion. It is a chat model, trained for:
- Sinhala conversation and writing;
- summarisation;
- : answering from documents you give it, with citation-numbered passages, and saying when the answer is not there;

The Sinhala tokenizer (custom built)

Most open LLMs use a byte-level BPE tokenizer with a GPT-2-style pre-tokenizer: a regex first cuts text into "words", and BPE merges are only learned inside those words. That regex defines a word as \p{L}+ (Unicode letters only).

The problem: Sinhala vowel signs (ා ි ී ු ෙ ් ං …) are Unicode combining marks (\p{M}), not letters, and the zero-width joiner (U+200D) used in conjuncts like ශ්‍රී and ප්‍ර is a format character. So the standard regex cuts every Sinhala word at every vowel sign. For example, ලංකාව is split into ල | ං | ක | ා | ව before BPE even runs, and no token can ever span a syllable. On top of that, each Sinhala character is 3 bytes in UTF-8, so the result is 2–3 tokens per character. A Sinhala sentence costs 5–10× more tokens than the same sentence in English.

What we built:

Sinhala-aware pre-tokenizer. The same GPT-2 regex, but a "word" is [\p{L}\p{M}‌‍]+: letters, combining marks, ZWNJ and ZWJ together.
text
's|'t|'re|'ve|'m|'ll|'d| ?[\p{L}\p{M}‌‍]+| ?\p{N}+| ?[^\s\p{L}\p{M}‌‍\p{N}]+|\s+(?!\S)|\s+
Text without combining marks (English, code, numbers) is split exactly as before, so English behaviour is unchanged.
Zero-width-joiner repair. Large web corpora (FineWeb-2, HPLT) silently drop the ZWJ, turning ශ්‍රී into ශ්රී. A normaliser restores it for 16 common conjunct clusters (rakaransaya ්‍ර, yansaya ්‍ය, …) before training, so the tokenizer and model learn the correct spellings.
Sinhala BPE. Trained a byte-level BPE (same 256-symbol byte alphabet as the base model) on 300k cleaned Sinhala documents (FineWeb-2) with the new pre-tokenizer.
Vocabulary extension, not replacement.
- The Sinhala merges are appended in order after the base model's 48,900 merges.
- Tokens that already exist, and pure-ASCII tokens, are skipped.
- All 49,152 original token IDs are unchanged, so the base model's knowledge stays valid.
This adds , for a total vocabulary of 63,344 (the embedding matrix is padded to 63,360 for GPU efficiency).

Result:

Table with columns: Text, Characters, SmolLM2 tokens, SinhalaLLM tokens
Text	Characters	SmolLM2 tokens	SinhalaLLM tokens
ශ්‍රී ලංකාවේ අගනුවර ශ්‍රී ජයවර්ධනපුර කෝට්ටේ වේ.	47	125	8
මම පාසල් යනවා.	14	36	4
ප්‍රභාසංශ්ලේෂණය	15	44	6

Whole words become single tokens: ' ලංකාවේ', ' ජයවර්ධනපුර', ' යනවා'. Rare long words split at sensible syllable boundaries: ප්‍ර | භා | සං | ශ් | ලේ | ෂණය.

Tokens per Sinhala character on held-out text:

Table with columns: Tokenizer, Tokens per char
Tokenizer	Tokens per char
SinhalaLLM	0.244
SinLlama (Llama-3 + Sinhala extension)	0.243
Gemma	0.836
Qwen2.5	1.447
SmolLM2	2.610

Using this tokenizer for other models

SmolLM2 family (135M, 360M, 1.7B). These share the same base tokenizer, so this tokenizer can be used directly: resize the embeddings and initialise the new rows as above.
Other byte-level BPE models (Llama 3, Qwen2.5, Mistral, …). Token IDs differ between model families, so the file cannot be dropped in as is. The method can be re-run on their tokenizer in minutes, on CPU, to produce an equivalent Sinhala extension:
- Sinhala-aware pre-tokenizer;
- ZWJ repair;
- BPE trained on Sinhala;
- merges appended without changing existing IDs;
- mean-of-pieces initialisation.
New models trained from scratch. The pre-tokenizer and the Sinhala merges can be used as the basis of a new Sinhala or Sinhala-English tokenizer.

The same pre-tokenizer fix applies to other Brahmic-script languages whose vowel signs are combining marks (Tamil, Hindi, Bengali and others).

How it was built

Sinhala-aware tokenizer: see above.
Embedding initialisation: each new token starts as the mean of the old byte-level pieces it replaces.
Continued pretraining on ~6.5B characters of cleaned Sinhala web text (FineWeb-2, HPLT 2.0, MADLAD-400, Wikipedia), with zero-width-joiner repair, language filtering and near-deduplication:
- Stage A: new embedding rows only.
- Stage B: full model, 4,500 steps × 131k tokens (~590M tokens), with 15% English (FineWeb-Edu) mixed in to keep English ability.
Instruction tuning (SFT) on ~40k Sinhala conversations: general chat, writing, RAG (answer from given passages or say the answer is not there), summarisation, and the model's identity. Answer-only loss.
Curriculum round on Sri Lankan school-level (Grades 6–11) Sinhala educational text, mixed with chat replay so chat ability is kept.
Clean-up round (v2.1) removing translation artefacts (e.g. leaked […පින්තූරය] placeholders) that some public Sinhala chat datasets contain. Answers with such artefacts: 30% → 0%.

Training: 1× RTX 5090, bf16, 8-bit AdamW, gradient checkpointing.

Evaluation

Sinhala language modelling and reading comprehension

Bits per character (lower = better) is comparable across models with different tokenizers. Belebele: 900 Sinhala reading-comprehension questions, 4 choices (25% = random).

Full numbers, including held-out Sinhala web text and generation speed:

Table with columns: Model, Size, bits/char FLORES ↓, bits/char web ↓, Belebele ↑, tokens/char ↓, Sinhala chars/s ↑, Run
Model	Size	bits/char FLORES ↓	bits/char web ↓	Belebele ↑	tokens/char ↓	Sinhala chars/s ↑	Run
SinhalaLLM v2.1	1.74B	1.475	1.430	34.8	0.244	422	2
SinhalaLLM v2	1.74B	1.476

Notes:

Belebele is length-normalised accuracy.
Two runs used the same script and GPU.
The quality numbers are deterministic. Speed varies about ±20% between runs (v2 vs v2.1 is essentially the same model), so compare speed within a run.
The iCIIT repo ships no tokenizer files, so it was run with the tokenizer of the Qwen2.5-1.5B model it was trained from.

RAG (answering from given passages)

60 unseen questions from SQAD. 40 have the answer in the passages; 20 do not.

Table with columns: v1, v2, v2.1
	v1	v2	v2.1
Refused although the answer was given ↓	35%	25%	12.5%
Answer contains the exact gold span ↑	15%	20%	20%
Correctly said "not in the passages" ↑	70%	30%	40%

Factual knowledge (closed-book)

44 short Sinhala fact questions (Sri Lankan geography, history, science, general knowledge), asked without any document. An answer counts as correct if it contains the right answer or an accepted variant. Two modes:

Few-shot Q&A (ප්‍රශ්නය: … පිළිතුර: …): fair for all models, including base models that are not chat models.
Chat: how people actually use SinhalaLLM.

Table with columns: Model, Size, Few-shot Q&A ↑, Chat ↑
Model	Size	Few-shot Q&A ↑	Chat ↑
SinhalaLLM v2.1	1.74B	13 / 44 (30%)	19 / 44 (43%)
SinhalaLLM v2	1.74B	13 / 44 (30%)	20 / 44 (45%)
SinhalaLLM v1	1.74B	14 / 44 (32%)	18 / 44 (41%)
Gemma-2B Sinhala (Udith-Sandaruwan)	2.5B	1 / 44 (2%)	–

Among small (≤3B) models, SinhalaLLM knows by far the most Sinhala facts: 13–19 correct vs 0–2 for the other small Sinhala and multilingual models.
The 4.7× larger SinLlama 8B knows much more (75%), as expected from its size and from Llama-3's much larger pretraining.
SinhalaLLM gives a 1.7B option that runs on a laptop.

The iCIIT checkpoint ships without tokenizer files, so it was run with the Qwen2.5-1.5B tokenizer it was trained from.

SinhalaLLM still gets many facts wrong from memory. Example of a confident but wrong answer:

Wrong fact example

The fluent Sinhala hides several errors:

John Loudon McAdam, the road engineer behind "macadam" roads, was Scottish, born in 1756, and died in 1836. The model invented the dates, the country and the "Great Western Power Station".
Even its own arithmetic does not add up: 1832 to 1905 is 73 years, not 72.

This is typical hallucination for a small model. For example, it says the Portuguese built Parakrama Samudra. The main reasons:

Model size. How many facts a model can store grows with its number of parameters. At 1.7B, there is room for fluent language but only limited world knowledge. This is true of every model this size, in any language.
Little Sinhala training data. It saw ~590M Sinhala tokens in continued pretraining. Large models see trillions of tokens, and the whole open Sinhala web is small and noisy. Many Sri Lankan facts appear only a few times.
Knowledge transfer from English is weak at this size. The base model learned most of its world knowledge in English. Small models cannot reliably connect that knowledge to Sinhala questions.
Chat data is partly machine-translated. It teaches fluency and format, not Sri Lankan facts.

Use RAG for facts. When the source text is provided, the model answers from it (see the RAG table above). This is the recommended way to use a model of this size for anything factual.

Future work

Scale up to 7B–15B parameters. Apply the same recipe to a stronger multilingual base model, for example Qwen2.5-7B/14B, Llama-3.1-8B, Gemma-2-9B or Mistral-Nemo-12B:
- Sinhala tokenizer extension;
- continued pretraining;
- chat, RAG and identity tuning.
Larger models store far more facts and transfer English knowledge to Sinhala much better; SinLlama 8B's 75% in the facts test above shows the gap. Combining that scale with our tokenizer method and our chat/RAG tuning is the main path to fixing factual accuracy.
More and better Sinhala data.
- 2–5B tokens of cleaned Sinhala;
- native (not translated) instruction data;
- question–answer pairs built from educational text.
Better grounding.
- A retrieval system over trusted sources (encyclopaedia, textbooks, government documents);

Use

Laptop (no GPU): download SinhalaLLM-v2.1-Q8_0.gguf or -Q4_K_M.gguf and run with llama.cpp or LM Studio. Recommended settings:

temperature 0.3, top-p 0.9, min-p 0.1, repeat penalty 1.15;
no system prompt: long system or tool prompts confuse a model this small.

bash

llama-server -m SinhalaLLM-v2.1-Q8_0.gguf -c 2048 --temp 0.3 --top-p 0.9 --min-p 0.1 --repeat-penalty 1.15

Transformers:

python

from transformers import AutoTokenizer, AutoModelForCausalLMtok = AutoTokenizer.from_pretrained("namindu94/SinhalaLLM-1.7B")model = AutoModelForCausalLM.from_pretrained("namindu94/SinhalaLLM-1.7B", dtype="bfloat16", device_map="auto")msgs = [{"role": "user", "content": "ශ්‍රී ලංකාවේ අගනුවර කුමක්ද?"}]ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)out = model.generate(ids, max_new_tokens=300, do_sample=True, temperature=0.3, top_p=0.9, repetition_penalty=1.15)print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

RAG prompt format (the format it was trained on):

text

පහත දී ඇති තොරතුරු පමණක් භාවිතා කර ප්‍රශ්නයට සිංහලෙන් පිළිතුරු දෙන්න. පිළිතුර එම තොරතුරුවල නොමැති නම්, එය පවසන්න.### තොරතුරු[1] <passage 1>[2] <passage 2>### ප්‍රශ්නය<question>

Limitations

Small model, limited world knowledge. It writes fluent Sinhala but often gets facts wrong from memory (history, dates, recent events); see Factual knowledge above. For factual use, give it the source text (RAG) and verify answers.
It sometimes answers when the passages don't contain the answer (see the RAG table). Add an application-level check for high-stakes use.
Weak at multi-step arithmetic and reasoning.
Spoken/colloquial Sinhala is understood, but replies are usually formal.
Trained context is 2,048 tokens.
Not for medical, legal or financial advice.

Acknowledgements

Built on SmolLM2 by Hugging Face. Thanks to the creators of the open Sinhala datasets listed above, and to llama.cpp for GGUF support.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.