TL;DR
v1 of my Gemma 4 12B Hindi fine-tune, trained on 12.5k Hindi instruction pairs. It meets its main goals:
no English words in brackets, and requested counts followed (7/7). Because much of the training data is
machine-translated, it also picked up some translation habits: extra length, a few artifacts, and occasional
factual slips. A 4-line system prompt on the base model currently scores higher (24 vs 20 fully correct out of 25).
v2 will fix this by training on the model's own clean Hindi answers (self-distillation).
Results (25 Hindi prompts)
Table with columns: Base + system prompt, This LoRA (temp 0.3) | Base + system prompt | This LoRA (temp 0.3) |
|---|
| Manual review: correct / partly wrong / wrong | 24 / 1 / 0 | 20 / 3 / 2 |
| English words in brackets | 0 | 0 |
| Requested list counts met | 7/7 | 7/7 |
| Visarga used as a colon ("हैः") | 2 | 45 |
| Average answer length | 219 chars | 628 chars |
| Chatbot filler ("नमस्ते! मुझे खुशी है…") | 0 | 5 answers |
Known issues (fixing in v2)
Most of the training data is machine-translated into Hindi, and the 12B base already writes better Hindi
than that data. So v1 picked up some of the data's habits:
- Translation artifacts: visarga (ः) instead of colons; "order" translated as "आदेश" (a command); lists missing their "1."
- Western content in Hindi: advised wearing a sweater during the Indian monsoon, missing mosquitoes and water-borne disease
- Factual slips: expanded UPI as "Universal Payments Interface" (it is Unified Payments Interface)
- Verbosity: about 3× longer than needed; 4 of 25 answers hit the token limit
- Foreign-script tokens at temperature 1.0: Russian and Chinese tokens mid-sentence. These disappear at temperature 0.7 and below
The same data improved my smaller models (0.6B–4B). At 12B, the base model is already stronger than the data.
For the best quality today
Base Gemma 4 12B with this system prompt:
हमेशा केवल शुद्ध हिंदी में उत्तर दें। कोष्ठक में अंग्रेज़ी शब्द न लिखें।जितना पूछा जाए, ठीक उतना ही दें। Markdown का उपयोग न करें।गणित या तर्क के प्रश्नों में अंतिम उत्तर से पहले एक-दो पंक्तियों में हल दिखाएँ।
The last line fixes the base model being too terse on maths. It was added after the evaluation.
Using this LoRA
The adapter is in the last-checkpoint folder.
from huggingface_hub import snapshot_downloadfrom unsloth import FastModel snapshot_download("pankajpandey-dev/gemma-4-12b-hindi-instruct-checkpoints", allow_patterns=["last-checkpoint/*"], local_dir="ckpt")model, tokenizer = FastModel.from_pretrained("ckpt/last-checkpoint", max_seq_length=2048, load_in_4bit=True)
Recommended settings: temperature 0.3 or lower (at 1.0 it can produce foreign-script tokens),
max new tokens around 400. Review outputs before using them in anything user-facing.
Training details
Table | |
|---|
| Base | unsloth/gemma-4-12b-it (Gemma 4 12B instruct), 4-bit QLoRA |
| Data | AI4Bharat indic-instruct-data-v0.1: anudesh Hindi (7,577) + 4,000 dolly Hindi; Cohere Aya Hindi (1,153). 12,478 examples after removing those over 1,024 tokens |
| LoRA | r=16, alpha=16, dropout 0, attention + MLP layers, vision layers frozen; 65.6M trainable parameters (0.55%) |
| Loss | Response-only (assistant tokens only), Gemma 4 chat template with empty thought channel |
| Optimiser | AdamW 8-bit, LR 1e-4, cosine, 20 warmup steps, weight decay 0.01 |
| Batch | 1 × 8 gradient accumulation, max length 1,024 |
| Schedule | 1 epoch, 1,560 steps |
| Precision | float32 compute (Gemma 4 can't train in fp16; the T4 has no bf16) |
| Hardware | Free Kaggle T4, 9 h 41 min |
| Final loss | 2.60 → ~1.2 plateau; 1.23 average over the run |
What's next
v2 with self-distillation: train on the 12B's own clean Hindi answers (base + system prompt, filtered)
instead of translated data, so the pure-Hindi behaviour is built in without losing quality.
Credits
Built with Unsloth and TRL. Data by AI4Bharat and Cohere Aya. 🙏