GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
IRedDragonICY
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
IRedDragonICY
Model Tree
google/gemma-4-E2B-it
Input Modalities
Output Modalities
Supported Functionality
Accuracy below is on the 1,200 LaguQA multiple-choice questions. The base model without training gets 24.1%. A guesser that has memorised the answer distribution but knows no song gets 32.1%, so anything under that line knows less about this book than the guesser does.
| Folder | Multiple choice (%) | Training data | Learning rate | Epochs | LoRA r | Alpha | Steps | Cost |
|---|---|---|---|---|---|---|---|---|
varian/lr4e4 | 61.0 | v1.4 | 0.0004 | 2 | 16 | 32 | 2622 | $2.19 |
(root) | 56.4 | v1.5 | 0.0004 | 2 | 16 | 32 | 2618 | $2.11 |
varian/epoch3 | 51.4 | v1.4 | 0.0002 | 3 | 16 | 32 | 3933 | $2.99 |
varian/r32 | 50.0 | v1.4 | 0.0002 | 2 | 32 | 64 | 2622 | $1.58 |
varian/epoch3-v15 | 46.2 | v1.5 | 0.0002 | 3 | 16 | 32 | 3927 | $5.13 |
varian/lr1e4 | 46.0 | v1.4 | 0.0001 | 2 | 16 | 32 | 2622 | $2.08 |
varian/r8 | 45.7 | v1.4 | 0.0002 | 2 | 8 | 16 | 2622 | $2.13 |
The varian/lr4e4 setting was run three times with different seeds and nothing
else changed: same training file by sha256, same learning rate, the same 2622
steps.
| seed | multiple choice |
|---|---|
| 1 | 61.0 |
| 2 | 52.3 |
| 3 | 56.7 |
Only seed 1 is published, as varian/lr4e4 in the table above. The other two
were measured to find the spread, not to be used.
Mean 56.7, standard deviation 4.3 points, range 8.7 points. That number is the limit for reading the whole table above: a gap under roughly nine points between two single runs cannot be interpreted.
This measurement changed two things that used to be written on this card. The
lr4e4 variant was once described as beating the v1.5 variant by 4.6 points,
but that gap is smaller than one standard deviation, and the means run the other
way, 56.7 against 56.4. The two are indistinguishable. The 61.0 is a lucky seed,
not a better setting.
Only the learning-rate difference survives the noise: 4e-4 against 2e-4 is worth about ten points, roughly two and a half standard deviations. LoRA rank shows no readable effect.
eval_loss cannot be used to compare seeds. The validation slice is held out
using the same seed as the training run, so each run is scored on different
data. The three seeds recorded 0.2385, 0.1741, and 0.1878, and the one with the
highest loss scored highest.
On the free-text track (1,002 questions, lenient accuracy), varian/lr4e4
seed 1 gets 76.6% and the root variant 76.5%, against 19.5% for the base model.
That track has not been measured for seeds 2 and 3, so its noise is unknown and
the 0.1-point gap means nothing.
The repository root holds the variant trained on laguqa_train.jsonl v1.5, the
file published as-is with the dataset. That is what loads when no subfolder is
named.
It was not chosen for having the highest score. Once run-to-run noise was
measured, this variant and varian/lr4e4 were indistinguishable, which left
reproducibility as the deciding factor. Anyone can retrain the root variant
straight from the published file.
varian/lr4e4 was trained on v1.4, which is not published as a file but can be
rebuilt. v1.4 and v1.5 differ in one column. v1.5 was built after composer
spellings were standardised on nine rows: six typos were merged into one
spelling, and three songs whose composer was written "NN" became empty. The
spelling printed in the book was not discarded, but kept in the
composer_printed column of laguqa.csv. Restoring the composer column to
that value and running the benchmark generator with the same seed produces an
identical file, sha256 f71707cd..., matching what that variant's
manifest.json records.
python
from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer dasar = "google/gemma-4-E2B-it"tok = AutoTokenizer.from_pretrained(dasar)model = AutoModelForCausalLM.from_pretrained(dasar, dtype="bfloat16")model = PeftModel.from_pretrained(model, "IRedDragonICY/LaguQA-Gemma4-E2B-LoRA") pesan = [{"role": "system", "content": "Kamu asisten yang menguasai lagu nasional dan lagu daerah " "Indonesia, termasuk notasi angka dan notasi ABC-nya."}, {"role": "user", "content": "Siapa pencipta lagu Gugur Bunga?"}]teks = tok.apply_chat_template(pesan, tokenize=False, add_generation_prompt=True)keluar = model.generate(**tok(teks, return_tensors="pt").to(model.device), max_new_tokens=192, do_sample=False)print(tok.decode(keluar[0], skip_special_tokens=True))The model answers in Indonesian, so the prompts above are kept in Indonesian.
Other variants load with subfolder:
python
model = PeftModel.from_pretrained(model, "IRedDragonICY/LaguQA-Gemma4-E2B-LoRA", subfolder="varian/lr4e4")Training on 20,944 examples drawn from a single book could damage what the model already knows. That was measured rather than assumed, using two Indonesian benchmarks unrelated to this project. Scoring is the same as above, and the base model was measured through the same code path on the same files.
| base | (root) | varian/lr4e4 | |
|---|---|---|---|
| IndoMMLU (2,000 questions, random guessing 23.4%) | 34.4 | 43.6 | 47.0 |
| IndoCulture (2,429 questions, random guessing 33.3%) | 58.7 | 54.6 | 56.9 |
Both are single seeds, so the gap between the two trained columns is subject to the same noise as the table above and is not worth reading. What is read here is the gap against the base model, which is far larger.
IndoMMLU goes up, not down. The gain is even across subject groups, including STEM, and is smallest on the language and regional-culture group. That pattern fits general improvement in Indonesian language modelling better than it fits transfer of song knowledge.
IndoCulture drops a few points, and the drop is larger on questions that are not province-specific than on those that are. What weakens appears to be the sense of which sentence sounds natural, not knowledge of regional custom.
Both numbers were measured under two different system-prompt conditions and point the same way, with under 1.5 points between conditions.
LoRA on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and
down_proj. Dropout 0.05, and 24,158,208 of 5,128,455,712 parameters (0.47%)
trained. Batch 4, gradient accumulation 4, 78 warmup steps, seed 1, bfloat16, no
quantisation. Run on a single NVIDIA L4 through modal.com. Library versions and
the training file's sha256 are recorded in each variant's manifest.json.
The training data is 20,944 conversation examples generated from 107 songs with templates rather than with an LLM, so the answers cannot be invented and the whole process can be repeated.
Free text used greedy decoding (do_sample=False), with no temperature, top-p,
or top-k, num_beams=1, and a maximum of 1024 new tokens.
Multiple choice was not scored from the letter the model writes. It was scored from the mean log-probability of each option's text, and the highest option was taken as the answer. Thinking mode follows the base model's chat template default and was not changed.
The adapter weights are released under Apache-2.0, the licence of
google/gemma-4-E2B-it. The data-generation and scoring code is Apache-2.0,
and the data is CC BY-NC 4.0.
The data comes from a 2025 songbook in number notation. The selection,
arrangement, and page layout of that book belong to its publisher, while each
song carries its own rights status. The details are in HAK-CIPTA.md in the
dataset repository.
bibtex
@misc{laguqa2026, title = {LaguQA: A Benchmark for Indonesian National and Regional Song Understanding in Large Language Models}, author = {Hendianto, Mohammad Farid}, year = {2026}, note = {Informatics, Universitas Ahmad Dahlan}, url = {https://huggingface.co/IRedDragonICY/LaguQA-Gemma4-E2B-LoRA}}from peft import PeftModelfrom transformers import AutoModelForCausalLM, AutoTokenizer dasar = "google/gemma-4-E2B-it"tok = AutoTokenizer.from_pretrained(dasar)model = AutoModelForCausalLM.from_pretrained(dasar, dtype="bfloat16")model = PeftModel.from_pretrained(model, "IRedDragonICY/LaguQA-Gemma4-E2B-LoRA") pesan = [{"role": "system", "content": "Kamu asisten yang menguasai lagu nasional dan lagu daerah " "Indonesia, termasuk notasi angka dan notasi ABC-nya."}, {"role": "user", "content": "Siapa pencipta lagu Gugur Bunga?"}]teks = tok.apply_chat_template(pesan, tokenize=False, add_generation_prompt=True)keluar = model.generate(**tok(teks, return_tensors="pt").to(model.device), max_new_tokens=192, do_sample=False)print(tok.decode(keluar[0], skip_special_tokens=True))model = PeftModel.from_pretrained(model, "IRedDragonICY/LaguQA-Gemma4-E2B-LoRA", subfolder="varian/lr4e4")@misc{laguqa2026, title = {LaguQA: A Benchmark for Indonesian National and Regional Song Understanding in Large Language Models}, author = {Hendianto, Mohammad Farid}, year = {2026}, note = {Informatics, Universitas Ahmad Dahlan}, url = {https://huggingface.co/IRedDragonICY/LaguQA-Gemma4-E2B-LoRA}}