Model Description
- Developed by: Saints Studios
- Model type: LlamaForCausalLM
- Language: Tumbuka (tum)
- License: Apache 2.0
Training Details
Training Data
The model was trained on a balanced corpus (approx. 50/50 split) of:
- Tumbuka Translated Gutenberg Corpus.
- Raw Tumbuka Text Repository.
Hyperparameters
- Architecture: 12 layers, 768 hidden size, 12 attention heads.
- Parameters: ~138 Million.
- Vocabulary Size: 16,384 (Byte-level BPE).
- Sequence Length: 512 tokens.
- Optimizer: AdamW with
learning_rate=3e-4 and weight_decay=0.1.
- Training Duration: 3 Epochs.
- Batch Size: 16 per device (with gradient accumulation).
How to use
from transformers import pipeline
generator = pipeline('text-generation', model='SaintsStudios/Mazgu_Small-T_130M')
prompt = "Mumutendeko "
print(generator(prompt, max_new_tokens=50))
Limitations and Biases
As a small-scale model trained on a niche dataset, Mazgu Small-T may exhibit hallucinations or repetitive behavior. It is intended primarily for research and further fine-tuning.
Example Outputs
--- Clean Inference Test (Native Generate) ---
Prompt: Pakwamba, nyengo zose tikugwiliskila ncito
Generated: Pakwamba , nyengo zose tiku gwiliskila ncito Pa webusayiti iyi pali nkhani yakukhwaskana na Project Gutenberg ndipo palije chinyake icho mungachita kweni kukhumba waka kuti muŵe ŵakukondwa . Kuti mu sankhe bwino zo chitika , muyenera kuganizira njira ziwiri zomwe zi mango gwirira ntchito ndiku wongolera . Mu mwezi wa chinkhonde na chitatu , pa zuŵa la khumi na limoza la mwezi uwo ,
Prompt: Chifukwa cha nyengo yiheni chomene, ŵalimi
Generated: Chifukwa cha nyengo yiheni chomene , ŵalimi Iyo wakati , ' Ine nizamu yowoya mazgu agha ! Ndipo wakati , Wonani , nkhumu pemphani kuti " K lo o de , uyo wali na vyaka pafupifupi 30 , wakwenera kuŵa musambizgi wa vya ma computer ndipo wali na mazaza pa nkhani ya umoyo . Para munthu wakuyowoya kuti , Inya , ndine mwana wane . Mu 18
Prompt: 1. Matenda gha ntchunga agho ghakupangika na
Generated: 1 . Matenda gha nt chu nga agho ghakupangika na " Yesu , ndine Chiuta ! Ŵasambiri ŵa sukulu ya sekondare ŵakakondwanga chomene chifukwa cha ivyo ŵakachitanga pa nyengo iyo . Ku zingilizga , kupanga na kukhozga vinthu Munthu yumoza pera , kweni palije munthu uyo wanga movwirani . Para munthu wakukhara mu charu ichi