NIVED2003

gemma-4-E4B-dora-prose-half2

README

License: apache-2.0

Model Details

Base Model: google/gemma-4-E4B
Tuning Method: DoRA (Weight-Decomposed Low-Rank Adaptation)
Rank ( $r$ ): 16
Alpha ( $α$ ): 32 ( $α / r = 2.0$ )
LoRA Dropout: 0.05
Target Modules: All 7 linear projection layers in the 42 text decoder layers:
- q_proj, k_proj, v_proj, o_proj (Self-Attention)
- gate_proj, up_proj, down_proj (MLP)
Trainable Parameters: 37,925,888 / 8,034,082,336 (0.472%)
Language Pair: Sanskrit (sa) $\leftrightarrow$ Hindi (hi) (Bidirectional)
Domain: Contemporary Sanskrit Prose & Parallel Benchmarks (Samanantar, Flores, IN22)

Evaluation & Training Dynamics

The model was trained for 2 epochs on 169,992 parallel sentence pairs and evaluated on 2,000 held-out validation pairs.

Validation Loss & Perplexity Trajectory

Table with columns: Milestone, Checkpoint Step, Epoch, Eval Loss (Cross-Entropy), Perplexity (PPL=eloss)
Milestone	Checkpoint Step	Epoch	Eval Loss (Cross-Entropy)	Perplexity ( $PPL = e loss$ )
Milestone 25%	5,312	0.50	`1.593`	4.92
Milestone 50%	10,625

The validation loss decreased monotonically across all four checkpoints down to 1.354 (Perplexity 3.87), confirming consistent convergence across independent splits.

Within-Domain Stability (Prose Half 1 vs. Prose Half 2)

Comparing both halves of the independent prose splits confirms high empirical consistency:

Table with columns: Metric, Prose Half 1, Prose Half 2, Absolute Delta (Δ), Relative Variance
Metric	Prose Half 1	Prose Half 2	Absolute Delta ( $Δ$ )	Relative Variance
Validation Loss	`1.326`	`1.354`	`0.028`	< 2.1%
Validation Perplexity	`3.77`	`3.87`

Training Hyperparameters

Hardware: 1x NVIDIA H100 80GB SXM5 GPU (Sapphire Rapids host)
Runtime: 10 hours 12 minutes (36,740s)
Throughput: 9.25 samples/sec (0.58 optimizer steps/sec)
Effective Batch Size: 16 (per_device_batch_size=4, gradient_accumulation_steps=4)
Optimizer: AdamW (weight_decay=0.01, max_grad_norm=1.0)
Learning Rate: 5e-5 with Cosine Annealing and 100 linear warmup steps
Sequence Length: 384 tokens (loss computed strictly on target tokens with -100 prompt masking)
Precision: bfloat16

How to Use

python

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLMfrom peft import PeftModel BASE_MODEL = "google/gemma-4-E4B"ADAPTER_REPO = "NIVED2003/gemma-4-E4B-dora-prose-half2" # 1. Load Tokenizer & Base Modeltokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)model = AutoModelForCausalLM.from_pretrained(    BASE_MODEL,    torch_dtype=torch.bfloat16,    device_map="auto") # 2. Load DoRA Adaptermodel = PeftModel.from_pretrained(model, ADAPTER_REPO)model.eval() # 3. Sanskrit -> Hindi Translationsanskrit_text = "भारतं विश्वस्य प्राचीनतमासु संस्कृतिषु अन्यतमा वर्तते।"prompt = f"Instruction: Translate the following contemporary Sanskrit text to Hindi.\nInput: {sanskrit_text}\nOutput: " inputs = tokenizer(prompt, return_tensors="pt").to("cuda") with torch.no_grad():    outputs = model.generate(        **inputs,        max_new_tokens=128,        temperature=0.3,        top_p=0.9,        do_sample=True,        eos_token_id=tokenizer.eos_token_id    ) translation = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)print("Hindi Translation:", translation) # 4. Hindi -> Sanskrit Translationhindi_text = "विज्ञान और तकनीक के क्षेत्र में नए शोध अत्यंत महत्वपूर्ण हैं।"prompt_reverse = f"Instruction: Translate the following contemporary Hindi text to Sanskrit.\nInput: {hindi_text}\nOutput: " inputs_rev = tokenizer(prompt_reverse, return_tensors="pt").to("cuda")with torch.no_grad():    outputs_rev = model.generate(        **inputs_rev,        max_new_tokens=128,        temperature=0.3,        top_p=0.9,        do_sample=True,        eos_token_id=tokenizer.eos_token_id    ) translation_sa = tokenizer.decode(outputs_rev[0][inputs_rev.input_ids.shape[1]:], skip_special_tokens=True)print("Sanskrit Translation:", translation_sa)

Intermediate Milestones Available

All intermediate milestone checkpoints are versioned in this repository:

checkpoints/step_25pct: 25% training milestone (step 5,312)
checkpoints/step_50pct: 50% training milestone (step 10,625)
checkpoints/step_75pct: 75% training milestone (step 15,938)
Root directory (.): Final fully-trained 100% adapter (step 21,250)

Citation & Architecture

bibtex

@inproceedings{liu2024dora,  title={DoRA: Weight-Decomposed Low-Rank Adaptation},  author={Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Khona, Pavlo and Shen, Sheng and Yen, Chen-Yu and Wang, Ting-Kuei and Chen, Kuan-Yu and Darve, Eric and Chen, Kwang-Ting},  booktitle={International Conference on Machine Learning (ICML)},  year={2024}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

NIVED2003

Model Tree

Base

google/gemma-4-E4B

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Base Model: google/gemma-4-E4B
Tuning Method: DoRA (Weight-Decomposed Low-Rank Adaptation)
Rank ( $r$ ): 16
Alpha ( $α$ ): 32 ( $α / r = 2.0$ )
LoRA Dropout: 0.05
Target Modules: All 7 linear projection layers in the 42 text decoder layers:
- q_proj, k_proj, v_proj, o_proj (Self-Attention)
- gate_proj, up_proj, down_proj (MLP)
Trainable Parameters: 37,925,888 / 8,034,082,336 (0.472%)
Language Pair: Sanskrit (sa) $\leftrightarrow$ Hindi (hi) (Bidirectional)
Domain: Contemporary Sanskrit Prose & Parallel Benchmarks (Samanantar, Flores, IN22)

Evaluation & Training Dynamics

The model was trained for 2 epochs on 169,992 parallel sentence pairs and evaluated on 2,000 held-out validation pairs.

Validation Loss & Perplexity Trajectory

Table with columns: Milestone, Checkpoint Step, Epoch, Eval Loss (Cross-Entropy), Perplexity (PPL=eloss)
Milestone	Checkpoint Step	Epoch	Eval Loss (Cross-Entropy)	Perplexity ( $PPL = e loss$ )
Milestone 25%	5,312	0.50	`1.593`	4.92
Milestone 50%	10,625

The validation loss decreased monotonically across all four checkpoints down to 1.354 (Perplexity 3.87), confirming consistent convergence across independent splits.

Within-Domain Stability (Prose Half 1 vs. Prose Half 2)

Comparing both halves of the independent prose splits confirms high empirical consistency:

Table with columns: Metric, Prose Half 1, Prose Half 2, Absolute Delta (Δ), Relative Variance
Metric	Prose Half 1	Prose Half 2	Absolute Delta ( $Δ$ )	Relative Variance
Validation Loss	`1.326`	`1.354`	`0.028`	< 2.1%
Validation Perplexity	`3.77`	`3.87`

Training Hyperparameters

Hardware: 1x NVIDIA H100 80GB SXM5 GPU (Sapphire Rapids host)
Runtime: 10 hours 12 minutes (36,740s)
Throughput: 9.25 samples/sec (0.58 optimizer steps/sec)
Effective Batch Size: 16 (per_device_batch_size=4, gradient_accumulation_steps=4)
Optimizer: AdamW (weight_decay=0.01, max_grad_norm=1.0)
Learning Rate: 5e-5 with Cosine Annealing and 100 linear warmup steps
Sequence Length: 384 tokens (loss computed strictly on target tokens with -100 prompt masking)
Precision: bfloat16

How to Use

python

import torchfrom transformers import AutoTokenizer, AutoModelForCausalLMfrom peft import PeftModel BASE_MODEL = "google/gemma-4-E4B"ADAPTER_REPO = "NIVED2003/gemma-4-E4B-dora-prose-half2" # 1. Load Tokenizer & Base Modeltokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)model = AutoModelForCausalLM.from_pretrained(    BASE_MODEL,    torch_dtype=torch.bfloat16,    device_map="auto") # 2. Load DoRA Adaptermodel = PeftModel.from_pretrained(model, ADAPTER_REPO)model.eval() # 3. Sanskrit -> Hindi Translationsanskrit_text = "भारतं विश्वस्य प्राचीनतमासु संस्कृतिषु अन्यतमा वर्तते।"prompt = f"Instruction: Translate the following contemporary Sanskrit text to Hindi.\nInput: {sanskrit_text}\nOutput: " inputs = tokenizer(prompt, return_tensors="pt").to("cuda") with torch.no_grad():    outputs = model.generate(        **inputs,        max_new_tokens=128,        temperature=0.3,        top_p=0.9,        do_sample=True,        eos_token_id=tokenizer.eos_token_id    ) translation = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)print("Hindi Translation:", translation) # 4. Hindi -> Sanskrit Translationhindi_text = "विज्ञान और तकनीक के क्षेत्र में नए शोध अत्यंत महत्वपूर्ण हैं।"prompt_reverse = f"Instruction: Translate the following contemporary Hindi text to Sanskrit.\nInput: {hindi_text}\nOutput: " inputs_rev = tokenizer(prompt_reverse, return_tensors="pt").to("cuda")with torch.no_grad():    outputs_rev = model.generate(        **inputs_rev,        max_new_tokens=128,        temperature=0.3,        top_p=0.9,        do_sample=True,        eos_token_id=tokenizer.eos_token_id    ) translation_sa = tokenizer.decode(outputs_rev[0][inputs_rev.input_ids.shape[1]:], skip_special_tokens=True)print("Sanskrit Translation:", translation_sa)

Intermediate Milestones Available

All intermediate milestone checkpoints are versioned in this repository:

checkpoints/step_25pct: 25% training milestone (step 5,312)
checkpoints/step_50pct: 50% training milestone (step 10,625)
checkpoints/step_75pct: 75% training milestone (step 15,938)
Root directory (.): Final fully-trained 100% adapter (step 21,250)

Citation & Architecture

bibtex

@inproceedings{liu2024dora,  title={DoRA: Weight-Decomposed Low-Rank Adaptation},  author={Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Khona, Pavlo and Shen, Sheng and Yen, Chen-Yu and Wang, Ting-Kuei and Chen, Kuan-Yu and Darve, Eric and Chen, Kwang-Ting},  booktitle={International Conference on Machine Learning (ICML)},  year={2024}}