gemma-4-e4b-sinhala-mmlu API & Inference Endpoint

Try the model

The validated runtime was vllm/vllm-openai:v0.32.0-cu129, with vLLM 0.32.0, PyTorch 2.13.0+cu129 and Transformers 5.19.0, on an A100 40 GB. requirements.txt records the package versions. Memory requirements depend on context length, batching and backend.

Download the published snapshot:

bash

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu --revision v2.0.0 --local-dir gemma-sinhala-mmlucd gemma-sinhala-mmlu

In the validated GPU environment, run the newly authored arithmetic example:

bash

python3 scripts/bakeoff_run.py \  --model gemma4_e4b --hf-id . \  --inputs examples --sets demo --out results_demo \  --modes think --samples 1 --greedy --context \  --think-budget 7168 --max-model-len 16384

Predictions are written to results_demo/gemma4_e4b/think_demo.jsonl. The example is a format demonstration, not an evaluation set. If the vLLM image raises a native torchcodec import error for this text-only task, remove the unused package with pip uninstall -y torchcodec, as in the validation environment.

Transformers example

python

import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizer model_id = "sinhala-mmlu/gemma-4-e4b-sinhala-mmlu"revision = "v2.0.0"tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)model = AutoModelForImageTextToText.from_pretrained(    model_id,    revision=revision,    dtype=torch.bfloat16,    device_map="auto",).eval() messages = [    {        "role": "system",        "content": (            "You are an expert teacher of the Sri Lankan national school curriculum, "            "fluent in Sinhala. Answer the multiple-choice question carefully. "            "You may reason in English. Finish with 'Answer: <number>'."        ),    },    {        "role": "user",        "content": (            "විෂය (Subject): Mathematics | ශ්‍රේණිය (Grade): 6\n\n"            "ප්‍රශ්නය (Question):\nදෙක සහ තුන එකතු කළ විට ලැබෙන අගය කීයද?\n\n"            "විකල්ප (Options):\n1. 4\n2. 5\n3. 6\n4. 7\n\n"            "Which option is correct? End your reply with 'Answer: <number>'."        ),    },]inputs = tokenizer.apply_chat_template(    messages,    enable_thinking=True,    add_generation_prompt=True,    return_dict=True,    return_tensors="pt",).to(model.device)with torch.inference_mode():    output = model.generate(**inputs, max_new_tokens=7168, do_sample=False)print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=False))

The output can contain reasoning-channel markers before the final Answer: N. For the exact MCQ prompt, parsing and forced-option scoring used in evaluation, use the supplied scripts/bakeoff_run.py. The shorter Transformers example above demonstrates loading and generation; it is not the complete benchmark harness.

Validation results

Measured on 27 October 2026 with the checkpoint in this repository, using the same benchmark code, questions and settings as v1.0.0: BF16, seed 1234, one greedy thinking sample, a 7,168-token budget, subject/grade metadata, and no option shuffling. Every checkpoint file's SHA-256 was verified before inference. The change column is paired over the same questions, with a bootstrap 95% interval.

Table with columns: Set, Questions, v1.0.0, v2.0.0, Change (95% interval)
Set	Questions	v1.0.0	v2.0.0	Change (95% interval)
Locked SinhalaMMLU Easy development subset	600	67.33%	68.00%	+0.67 (−1.83, +3.50)
Global-MMLU Sinhala harder-domain proxy	1,000	74.30%	76.80%	+2.50 (+0.50, +4.60)
Equal-weight mean of those two sets	1,600 total	70.82%	72.40%	+1.58 (−0.15, +3.22)

This is not the official SinhalaMMLU Easy/Medium/Hard competition score. The official development set contains only Easy (grade 6–8) questions, and the official Medium/Hard test answers are not public. The Global-MMLU subset is a proxy, not the official Hard test. The locked 600 Easy questions were excluded from training by ID; a normalized question/options check found one locked question that the official development set also contains under another ID, and that copy was in training.

Supplementary curriculum sets, generated from held-out textbook passages (grades 6–8 / 9–11 / Advanced Level / aesthetic subjects):

Table with columns: Set, Questions, v1.0.0, v2.0.0, Change (95% interval)
Set	Questions	v1.0.0	v2.0.0	Change (95% interval)
Curriculum Easy	195	87.18%	89.23%	+2.05 (−1.03, +5.64)
Curriculum Medium	282	82.62%	81.56%	−1.06 (−4.61, +2.48)
Curriculum Hard	267	87.64%	87.64%	0.00 (−3.37, +3.37)
Curriculum Aesthetic	282	83.69%	85.46%	+1.77 (−1.42, +4.61)

These questions were written by Codex from passages held out of training. They are easier than real exam questions and may favor models trained on similar generated questions, so they are excluded from the primary average.

Reasoning matters more for v2.0.0 than for v1.0.0: without thinking, Gemma native direct-answer scoring averaged 48.63% on the two primary sets (v1.0.0: 51.37%), and a raw completion prompt averaged 35.80% (v1.0.0: 36.10%). Use the reasoning template.

2,625 of the 2,626 reasoning outputs closed their thinking block and produced a parseable answer; one reached the 7,168-token cap. Generation plus final option scoring took 578.5 seconds in the measured A100 batch run, excluding startup, weight transfer and verification. This is not single-request latency. Aggregate results, the paired comparison with v1.0.0 and input fingerprints are in evaluation/metrics.json and evaluation/input_fingerprints.json. Evaluation questions and answers are not distributed here.

The shared-task submission recipe used 16 sampled votes, option shuffling, subject/grade context and a 7,168-token budget, a different configuration from the one-sample greedy numbers above. The included evaluator supports it with --samples 16 --shuffle --context --think-budget 7168, without --greedy.

Predict one JSON question file

hf download only downloads files. It does not read a question file, run inference or create predictions. Its --local-dir argument is a destination directory, not the path to your questions.

For example, if your questions are already in testing-flow/ques.json, run the following from your project directory in the Linux CUDA GPU environment described above. The first command downloads the full checkpoint and scripts into a separate model directory; the second command runs inference and writes the answer JSONL.

bash

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  --revision main --local-dir testing-flow/modelpython3 testing-flow/model/scripts/predict_json.py \  testing-flow/ques.json testing-flow/predictions.jsonl \  --model testing-flow/model

The input must be a JSON array of question objects with q_no, subject, category, question and exactly four string choices. Each question needs level or metadata.difficulty (easy, medium or hard); use --level hard, for example, as a fallback if the file has no difficulty metadata. An answer key is not needed, and any supplied answers are excluded from the model input.

The command uses the existing evaluator with 16 thinking samples, option shuffling, subject/grade context and the 7,168-token thinking budget. It writes one row per input question to testing-flow/predictions.jsonl, containing only q_no, subject, category, level and the string answer. Detailed model outputs are kept in testing-flow/predictions.run/. Any input size is supported; a 30-question file produces 30 prediction rows. A subset alone is not a complete official test submission.

Default inference requires the vLLM/CUDA runtime; an experimental 4-bit CUDA alternative is described below. Neither backend runs model inference on macOS. To check and prepare a file on a machine without that runtime, add --prepare-only. This validates the questions and writes the intermediate input without loading the model or writing predictions:

bash

python3 testing-flow/model/scripts/predict_json.py \  testing-flow/ques.json testing-flow/predictions.jsonl --prepare-only

Experimental Colab T4 mode

The BF16 vLLM recipe above was validated on an A100 40 GB. A T4 has less memory and needs a different setup. --backend transformers-4bit provides an experimental, sequential alternative using Transformers, bitsandbytes NF4 weights and FP16 computation. Bitsandbytes supports NF4 on T4-class hardware; the memory fit, numerical behavior and accuracy of this checkpoint on a T4 have not been GPU-tested. This mode uses one greedy thinking sample with the original option order; it does not reproduce the published BF16 validation numbers or the 16-vote submission recipe.

In Colab, first select a T4 GPU runtime. nvidia-smi must show the attached GPU. Keep Colab's preinstalled CUDA-enabled PyTorch, and install the separate small-GPU requirements; vLLM is not used by this mode.

python

!nvidia-smi%pip install -q huggingface_hub!hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  scripts/predict_json.py scripts/predict_transformers.py scripts/bakeoff_run.py \  requirements-colab-t4.txt --revision main --local-dir /content/runner%pip install -q -r /content/runner/requirements-colab-t4.txt

Upload your question file as /content/ques.json. Then run the command below; it downloads the checkpoint weights from the Hub as needed and writes /content/predictions.jsonl. Reupload the file and rerun setup if switching runtimes created a fresh session. If Colab requests a session restart after installation, restart before running inference.

python

!python -u /content/runner/scripts/predict_json.py \  /content/ques.json /content/predictions.jsonl \  --model sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  --backend transformers-4bit --samples 1 --think-budget 7168

This path processes one question at a time. It parses the answer after the thinking block, falling back to scores for options 1–4 if the answer is missing or the thinking budget is exhausted. Non-finite fallback scores raise an error. Detailed responses are saved in /content/predictions.run/; a complete submission JSONL is written only after all questions finish successfully. Quantization and the changed decoding recipe can affect predictions, and generation may be slow.

Generate the Codabench submission

The SinhalaMMLU shared task on Codabench scores a ZIP containing predictions.jsonl against the organizers' private answer key. The public shared-task test release contains 2,238 questions: 642 Easy, 1,111 Medium and 485 Hard. It has no gold answers at any level. The labeled Easy development data used for validation above is a separate dataset.

This checkpoint already produced submission 08 in the required format. Each JSONL line contains the original test identifiers and one string answer, "1", "2", "3" or "4":

json

{"q_no":1,"subject":"Arts","category":"humanities","level":"easy","answer":"2"}

Keep q_no, subject, category and level exactly as supplied by the test data. A question number alone is not a unique identifier across subjects and levels. The internal evaluator output uses id and pred and includes diagnostic fields; the exporter below writes only the five submission fields, in test-file order, with one prediction per question.

From the checkpoint directory created in Try the model, download the submission helpers added on main after the original v2.0.0 weight release:

bash

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  README.md SHA256SUMS scripts/build_test_inputs.py scripts/export_submission.py \  --revision main --local-dir .

Obtain the questions from the official test dataset and prepare the evaluator input. The revision is pinned to the exact test release used for submission 08; the helpers do not download or infer any answer key.

bash

hf download SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset \  --type dataset --revision 56fb2da50cfcdf0a22bfaee179370eef170fdc98 \  --include '*.json' --local-dir shared_task_testpython3 scripts/build_test_inputs.py shared_task_test inputs/test_56fb.jsonl

In the validated GPU environment described above, generate predictions with the submission recipe, then export the ZIP:

bash

python3 scripts/bakeoff_run.py \  --model gemma4_e4b --hf-id . \  --inputs inputs --sets test_56fb --out results_test \  --modes think --samples 16 --shuffle --context \  --think-budget 7168 --max-model-len 16384python3 scripts/export_submission.py \  inputs/test_56fb.jsonl results_test/gemma4_e4b/think_test_56fb.jsonl \  submission --k 16

Upload submission/predictions.zip to Codabench. It contains only predictions.jsonl at the archive root. The exporter checks that all 2,238 test questions have predictions, that their submission keys are unique, and that each answer is in "1"–"4".

Retain the native thinking template and the full thinking budget when reproducing this recipe. The model can generate English reasoning and a final Answer: N; the evaluator parses the final answer, uses forced-option scoring if necessary, maps shuffled options back to their original positions, and combines the 16 votes. Only the resulting option number is written to the submission's answer field. A short, direct-answer demonstration prompt uses a different inference configuration.

Because the test input has no gold labels, the local evaluator reports accuracy: null. Official correctness is established by Codabench's hidden-label scoring; generating a valid ZIP does not establish an accuracy score. Sampling and GPU numerics can change some predictions when rerunning the recipe.

How this checkpoint was built

Sinhala curriculum material from public textbook, teacher-guide and syllabus sources (grades 6–13) was cleaned and converted to Unicode where needed, then split into passages.
Curriculum MCQs were written from the passages by open Gemma 4 26B-A4B and by Codex (gpt-6.1-sol). The Codex-written questions were checked blind by Codex or Claude Sonnet.
Reasoning traces were written without the source passage and without the answer key, then kept only if their final answer matched the key:
- 11,334 traces from open Gemma 4 26B-A4B and 31B teachers, using the student's exact prompt (the v1.0.0 data plus a second Gemma round);
- 9,536 traces from closed models: OpenAI Codex CLI gpt-6.1-sol (9,027), xAI Grok CLI grok-4.7-build-fast (397) and Anthropic Claude Code claude-opus-5-5 (112). They ran with tools and web search disabled (an early Codex batch could still reach a server-side search feature until it was switched off; traces containing URLs were removed) and wrote English bullet-point reasoning in the Gemma style. A trace was kept only if it reached the keyed answer, its reasoning ended on the same option, it had 120–900 words and it did not mention a passage, answer key or URL. Each question has at most one trace per model and two closed-model traces.
The final training file contained 20,870 traces over 14,231 unique questions: 4,841 at grades 6–8, 8,845 at grades 9–11 and 7,184 at Advanced Level. It includes 930 SinhalaMMLU development questions outside the locked 600-question evaluation subset; the rest are generated curriculum questions.
LoRA rank/alpha 64, dropout 0.05, all language-model linear layers, two epochs (2,320 optimizer steps on two A100 40 GB GPUs), learning rate 5e-5, maximum sequence length 3,072, micro-batch 1 per GPU and 16,384 tokens per batch. Vision/audio towers were not fine-tuned.
The adapter was merged and the weights interpolated as W_final = W_base + 0.5 * (W_distilled - W_base).

Closed models were used only to create training data. Only one merged model is loaded for inference. There is no retrieval, tool use or teacher-model call in the inference path.

Continue training

See TRAINING.md for a new LoRA run, dataset schema, checkpointing and merging. The full weights can also be used with another compatible fine-tuning framework.

This is a model-weight checkpoint, not an exact training-resume checkpoint: it does not contain the original optimizer/scheduler state or the unmerged pre-interpolation adapter. Further training starts with a new optimizer from the published merged weights. The supplied script can save resumable state for your own new run.

Scope and limitations

The evaluated specialization is text-based Sinhala MCQs. General chat, vision and audio quality were not evaluated for this derivative; retained multimodal weights come from the base model.
Responses can be incorrect. English reasoning is expected even when the question is in Sinhala.
The validation numbers are for this exact release and inference configuration; they do not transfer automatically to a further fine-tune.
Part of the training data was written by closed models (see above). Check those providers' terms before reusing this model or its outputs to build other models.
Original training and evaluation datasets are not part of this repository and retain their respective licenses and access conditions. Obtain suitable data from its original sources for your own training.

Base-model credit: Google/Google DeepMind, google/gemma-4-E4B-it. This is a community derivative maintained by the sinhala-mmlu organization, not an official Google release.

Try the model

Download the published snapshot:

bash

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu --revision v2.0.0 --local-dir gemma-sinhala-mmlucd gemma-sinhala-mmlu

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu --revision v2.0.0 --local-dir gemma-sinhala-mmlucd gemma-sinhala-mmlu

In the validated GPU environment, run the newly authored arithmetic example:

bash

python3 scripts/bakeoff_run.py \  --model gemma4_e4b --hf-id . \  --inputs examples --sets demo --out results_demo \  --modes think --samples 1 --greedy --context \  --think-budget 7168 --max-model-len 16384

python3 scripts/bakeoff_run.py \  --model gemma4_e4b --hf-id . \  --inputs examples --sets demo --out results_demo \  --modes think --samples 1 --greedy --context \  --think-budget 7168 --max-model-len 16384

Transformers example

python

import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizer model_id = "sinhala-mmlu/gemma-4-e4b-sinhala-mmlu"revision = "v2.0.0"tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)model = AutoModelForImageTextToText.from_pretrained(    model_id,    revision=revision,    dtype=torch.bfloat16,    device_map="auto",).eval() messages = [    {        "role": "system",        "content": (            "You are an expert teacher of the Sri Lankan national school curriculum, "            "fluent in Sinhala. Answer the multiple-choice question carefully. "            "You may reason in English. Finish with 'Answer: <number>'."        ),    },    {        "role": "user",        "content": (            "විෂය (Subject): Mathematics | ශ්‍රේණිය (Grade): 6\n\n"            "ප්‍රශ්නය (Question):\nදෙක සහ තුන එකතු කළ විට ලැබෙන අගය කීයද?\n\n"            "විකල්ප (Options):\n1. 4\n2. 5\n3. 6\n4. 7\n\n"            "Which option is correct? End your reply with 'Answer: <number>'."        ),    },]inputs = tokenizer.apply_chat_template(    messages,    enable_thinking=True,    add_generation_prompt=True,    return_dict=True,    return_tensors="pt",).to(model.device)with torch.inference_mode():    output = model.generate(**inputs, max_new_tokens=7168, do_sample=False)print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=False))

import torchfrom transformers import AutoModelForImageTextToText, AutoTokenizer model_id = "sinhala-mmlu/gemma-4-e4b-sinhala-mmlu"revision = "v2.0.0"tokenizer = AutoTokenizer.from_pretrained(model_id, revision=revision)model = AutoModelForImageTextToText.from_pretrained(    model_id,    revision=revision,    dtype=torch.bfloat16,    device_map="auto",).eval() messages = [    {        "role": "system",        "content": (            "You are an expert teacher of the Sri Lankan national school curriculum, "            "fluent in Sinhala. Answer the multiple-choice question carefully. "            "You may reason in English. Finish with 'Answer: <number>'."        ),    },    {        "role": "user",        "content": (            "විෂය (Subject): Mathematics | ශ්‍රේණිය (Grade): 6\n\n"            "ප්‍රශ්නය (Question):\nදෙක සහ තුන එකතු කළ විට ලැබෙන අගය කීයද?\n\n"            "විකල්ප (Options):\n1. 4\n2. 5\n3. 6\n4. 7\n\n"            "Which option is correct? End your reply with 'Answer: <number>'."        ),    },]inputs = tokenizer.apply_chat_template(    messages,    enable_thinking=True,    add_generation_prompt=True,    return_dict=True,    return_tensors="pt",).to(model.device)with torch.inference_mode():    output = model.generate(**inputs, max_new_tokens=7168, do_sample=False)print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=False))

Validation results

Table with columns: Set, Questions, v1.0.0, v2.0.0, Change (95% interval)
Set	Questions	v1.0.0	v2.0.0	Change (95% interval)
Locked SinhalaMMLU Easy development subset	600	67.33%	68.00%	+0.67 (−1.83, +3.50)
Global-MMLU Sinhala harder-domain proxy	1,000	74.30%	76.80%	+2.50 (+0.50, +4.60)
Equal-weight mean of those two sets	1,600 total	70.82%	72.40%	+1.58 (−0.15, +3.22)

Supplementary curriculum sets, generated from held-out textbook passages (grades 6–8 / 9–11 / Advanced Level / aesthetic subjects):

Table with columns: Set, Questions, v1.0.0, v2.0.0, Change (95% interval)
Set	Questions	v1.0.0	v2.0.0	Change (95% interval)
Curriculum Easy	195	87.18%	89.23%	+2.05 (−1.03, +5.64)
Curriculum Medium	282	82.62%	81.56%	−1.06 (−4.61, +2.48)
Curriculum Hard	267	87.64%	87.64%	0.00 (−3.37, +3.37)
Curriculum Aesthetic	282	83.69%	85.46%	+1.77 (−1.42, +4.61)

Predict one JSON question file

hf download only downloads files. It does not read a question file, run inference or create predictions. Its --local-dir argument is a destination directory, not the path to your questions.

bash

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  --revision main --local-dir testing-flow/modelpython3 testing-flow/model/scripts/predict_json.py \  testing-flow/ques.json testing-flow/predictions.jsonl \  --model testing-flow/model

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  --revision main --local-dir testing-flow/modelpython3 testing-flow/model/scripts/predict_json.py \  testing-flow/ques.json testing-flow/predictions.jsonl \  --model testing-flow/model

bash

python3 testing-flow/model/scripts/predict_json.py \  testing-flow/ques.json testing-flow/predictions.jsonl --prepare-only

python3 testing-flow/model/scripts/predict_json.py \  testing-flow/ques.json testing-flow/predictions.jsonl --prepare-only

Experimental Colab T4 mode

python

!nvidia-smi%pip install -q huggingface_hub!hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  scripts/predict_json.py scripts/predict_transformers.py scripts/bakeoff_run.py \  requirements-colab-t4.txt --revision main --local-dir /content/runner%pip install -q -r /content/runner/requirements-colab-t4.txt

!nvidia-smi%pip install -q huggingface_hub!hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  scripts/predict_json.py scripts/predict_transformers.py scripts/bakeoff_run.py \  requirements-colab-t4.txt --revision main --local-dir /content/runner%pip install -q -r /content/runner/requirements-colab-t4.txt

python

!python -u /content/runner/scripts/predict_json.py \  /content/ques.json /content/predictions.jsonl \  --model sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  --backend transformers-4bit --samples 1 --think-budget 7168

!python -u /content/runner/scripts/predict_json.py \  /content/ques.json /content/predictions.jsonl \  --model sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  --backend transformers-4bit --samples 1 --think-budget 7168

Generate the Codabench submission

This checkpoint already produced submission 08 in the required format. Each JSONL line contains the original test identifiers and one string answer, "1", "2", "3" or "4":

json

{"q_no":1,"subject":"Arts","category":"humanities","level":"easy","answer":"2"}

{"q_no":1,"subject":"Arts","category":"humanities","level":"easy","answer":"2"}

From the checkpoint directory created in Try the model, download the submission helpers added on main after the original v2.0.0 weight release:

bash

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  README.md SHA256SUMS scripts/build_test_inputs.py scripts/export_submission.py \  --revision main --local-dir .

hf download sinhala-mmlu/gemma-4-e4b-sinhala-mmlu \  README.md SHA256SUMS scripts/build_test_inputs.py scripts/export_submission.py \  --revision main --local-dir .

bash

hf download SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset \  --type dataset --revision 56fb2da50cfcdf0a22bfaee179370eef170fdc98 \  --include '*.json' --local-dir shared_task_testpython3 scripts/build_test_inputs.py shared_task_test inputs/test_56fb.jsonl

hf download SinhalaMMLU-SharedTask/Sinhala-MMLU-SharedTask-Test-Dataset \  --type dataset --revision 56fb2da50cfcdf0a22bfaee179370eef170fdc98 \  --include '*.json' --local-dir shared_task_testpython3 scripts/build_test_inputs.py shared_task_test inputs/test_56fb.jsonl

In the validated GPU environment described above, generate predictions with the submission recipe, then export the ZIP:

bash

python3 scripts/bakeoff_run.py \  --model gemma4_e4b --hf-id . \  --inputs inputs --sets test_56fb --out results_test \  --modes think --samples 16 --shuffle --context \  --think-budget 7168 --max-model-len 16384python3 scripts/export_submission.py \  inputs/test_56fb.jsonl results_test/gemma4_e4b/think_test_56fb.jsonl \  submission --k 16

python3 scripts/bakeoff_run.py \  --model gemma4_e4b --hf-id . \  --inputs inputs --sets test_56fb --out results_test \  --modes think --samples 16 --shuffle --context \  --think-budget 7168 --max-model-len 16384python3 scripts/export_submission.py \  inputs/test_56fb.jsonl results_test/gemma4_e4b/think_test_56fb.jsonl \  submission --k 16

How this checkpoint was built

Sinhala curriculum material from public textbook, teacher-guide and syllabus sources (grades 6–13) was cleaned and converted to Unicode where needed, then split into passages.
Curriculum MCQs were written from the passages by open Gemma 4 26B-A4B and by Codex (gpt-6.1-sol). The Codex-written questions were checked blind by Codex or Claude Sonnet.
Reasoning traces were written without the source passage and without the answer key, then kept only if their final answer matched the key:
- 11,334 traces from open Gemma 4 26B-A4B and 31B teachers, using the student's exact prompt (the v1.0.0 data plus a second Gemma round);
- 9,536 traces from closed models: OpenAI Codex CLI gpt-6.1-sol (9,027), xAI Grok CLI grok-4.7-build-fast (397) and Anthropic Claude Code claude-opus-5-5 (112). They ran with tools and web search disabled (an early Codex batch could still reach a server-side search feature until it was switched off; traces containing URLs were removed) and wrote English bullet-point reasoning in the Gemma style. A trace was kept only if it reached the keyed answer, its reasoning ended on the same option, it had 120–900 words and it did not mention a passage, answer key or URL. Each question has at most one trace per model and two closed-model traces.
The final training file contained 20,870 traces over 14,231 unique questions: 4,841 at grades 6–8, 8,845 at grades 9–11 and 7,184 at Advanced Level. It includes 930 SinhalaMMLU development questions outside the locked 600-question evaluation subset; the rest are generated curriculum questions.
LoRA rank/alpha 64, dropout 0.05, all language-model linear layers, two epochs (2,320 optimizer steps on two A100 40 GB GPUs), learning rate 5e-5, maximum sequence length 3,072, micro-batch 1 per GPU and 16,384 tokens per batch. Vision/audio towers were not fine-tuned.
The adapter was merged and the weights interpolated as W_final = W_base + 0.5 * (W_distilled - W_base).

Closed models were used only to create training data. Only one merged model is loaded for inference. There is no retrieval, tool use or teacher-model call in the inference path.

Continue training

See TRAINING.md for a new LoRA run, dataset schema, checkpointing and merging. The full weights can also be used with another compatible fine-tuning framework.

Scope and limitations

The evaluated specialization is text-based Sinhala MCQs. General chat, vision and audio quality were not evaluated for this derivative; retained multimodal weights come from the base model.
Responses can be incorrect. English reasoning is expected even when the question is in Sinhala.
The validation numbers are for this exact release and inference configuration; they do not transfer automatically to a further fine-tune.
Part of the training data was written by closed models (see above). Check those providers' terms before reusing this model or its outputs to build other models.
Original training and evaluation datasets are not part of this repository and retain their respective licenses and access conditions. Obtain suitable data from its original sources for your own training.

Base-model credit: Google/Google DeepMind, google/gemma-4-E4B-it. This is a community derivative maintained by the sinhala-mmlu organization, not an official Google release.

gemma-4-e4b-sinhala-mmlu

README

Try the model

Transformers example

Validation results

Predict one JSON question file

Experimental Colab T4 mode

Generate the Codabench submission

How this checkpoint was built

Continue training

Scope and limitations

Explore FriendliAI today

README

Try the model

Transformers example

Validation results

Predict one JSON question file

Experimental Colab T4 mode

Generate the Codabench submission

How this checkpoint was built

Continue training

Scope and limitations