spitfire4794

Alo-70m

README

License: apache-2.0

Model Summary

Alo-70M is the instruction-tuned version of the ultra-lightweight 69-million parameter Bengali language model, Alo-70M-Base. Built on a scaled-down LLaMA architecture, it is designed to act as a highly efficient, edge-deployable localized AI assistant.

Fine-tuned on a curated dataset of instruction-response pairs using the ChatML format, Alo-70M is aligned for tasks such as summarization, entity extraction, text editing, and question answering in native Bengali. Despite its compact footprint, it offers a viable path for edge AI deployment on standard CPUs and mobile hardware.

Developer: Fahad Hossain
Language: Bengali (Bangla)
Model Type: Causal Language Model (Instruction-Tuned Autoregressive Transformer)
Parameter Count: 69 Million
License: Apache 2.0

Base Model: spitfire4794/Alo-70M-Base
Alignment Dataset: spitfire4794/Bangla-SFT-50k
Tokenizer: spitfire4794/beng_bpe

Usage

Alo-70M was trained using the ChatML template. The chat template is built directly into the Jinja template of the tokenizer (spitfire4794/beng_bpe). You can leverage it using Hugging Face's apply_chat_template interface:

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "spitfire4794/Alo-70M"

# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Define the instruction in ChatML format
messages = [
    {"role": "user", "content": "নিচের অনুচ্ছেদটি সংক্ষেপে সারসংক্ষেপ করুন: [এখানে আপনার টেক্সট লিখুন]"}
]

# Apply the pre-configured ChatML template
inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

# Generate text
outputs = model.generate(
    **inputs, 
    max_new_tokens=150, 
    repetition_penalty=1.1,
    do_sample=True,
    temperature=0.6,
    top_p=0.9
)

# Decode response (omitting user prompt)
response = outputs[0][inputs.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

Supervised Fine-Tuning (SFT) Details

Alo-70M was aligned using a curated subset of the Bangla-SFT-50k dataset formatted using ChatML.

Dataset Pruning: Initial SFT experiments revealed that forcing a sub-100M parameter model to learn complex markdown syntax/tables caused severe representation crowding. Thus, the 12,517-sample Structured Formatting category was excluded. The final active training mixture consisted of 37,536 aligned pairs.
Engineering Properties: The training data strictly forbade conversational prefaces (e.g., "নিশ্চয়ই, আমি এটি করে দিচ্ছি") so that responses begin immediately with the target output, optimizing inference speeds.
Hardware: NVIDIA T4 and L4 GPUs.
Hyperparameters:
- Optimizer: Fused AdamW (adamw_torch_fused) with $β 1 = 0.9, β$

Model Architecture details

Like its base model, Alo-70M utilizes a parameter-efficient architecture:

Layers: 12
Hidden Dimension ( $d m o d e l $ ): 512 | Intermediate FFN: 1408
Attention: Grouped-Query Attention (GQA) with 8 query heads / 4 KV heads.
Positional Embeddings: RoPE (Base freq: 10,000)
Word Embeddings: Untied (tie_word_embeddings = False).
Context Window: 1024 tokens.

Evaluation Results

The model was evaluated zero-shot across Bengali reasoning and knowledge benchmarks (continuation-based log-probability evaluation):

Table with columns: Benchmark, Alo-70M (SFT), Alo-70M-Base, Gemma-3-270M-IT, TigerLLM-1B-IT
Benchmark	Alo-70M (SFT)	Alo-70M-Base	Gemma-3-270M-IT	TigerLLM-1B-IT
bangla_mmlu_bn	26.29%	26.31%	26.81%	27.66%
bangla_commonsenseqa_bn	25.88%	28.42%	22.77%	25.14%
indicbench_arc_bn_challenge	24.15%	22.70%	25.34%

Note: The 69M instruction-tuned model outperforms the larger Gemma-3-270M-IT baseline on tasks like CommonsenseQA and PIQA.

Limitations and Biases

Alignment Tax (Catastrophic Forgetting): While SFT successfully aligned the model for text generation stability and instruction following, it introduced a measurable degradation in pure zero-shot reasoning compared to the Base model (e.g., dropping from 28.42% to 25.88% on CommonsenseQA). This happens because applying instructions to a highly capacity-constrained 69M model over-indexes weights toward output formatting at the expense of some pre-trained logical representations.
Knowledge Retrieval: With under 100M parameters, the model physically lacks the capacity to serve as a comprehensive encyclopedic knowledge base. It is better suited for text processing tasks (editing, summarizing) than fact-retrieval.
Context Length: The model is optimized for a 1024-token context window. Prompts exceeding this length will be truncated or result in degraded quality.

Citation

Technical paper out soon.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

spitfire4794

Model Tree

Base

spitfire4794/Alo-70m-Base

Fine-tuned

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Container

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Summary

Developer: Fahad Hossain
Language: Bengali (Bangla)
Model Type: Causal Language Model (Instruction-Tuned Autoregressive Transformer)
Parameter Count: 69 Million
License: Apache 2.0

Base Model: spitfire4794/Alo-70M-Base
Alignment Dataset: spitfire4794/Bangla-SFT-50k
Tokenizer: spitfire4794/beng_bpe

Usage

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "spitfire4794/Alo-70M"

# Load the custom Bengali BPE tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Define the instruction in ChatML format
messages = [
    {"role": "user", "content": "নিচের অনুচ্ছেদটি সংক্ষেপে সারসংক্ষেপ করুন: [এখানে আপনার টেক্সট লিখুন]"}
]

# Apply the pre-configured ChatML template
inputs = tokenizer.apply_chat_template(
    messages, 
    add_generation_prompt=True, 
    return_tensors="pt"
).to(model.device)

# Generate text
outputs = model.generate(
    **inputs, 
    max_new_tokens=150, 
    repetition_penalty=1.1,
    do_sample=True,
    temperature=0.6,
    top_p=0.9
)

# Decode response (omitting user prompt)
response = outputs[0][inputs.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))

Supervised Fine-Tuning (SFT) Details

Alo-70M was aligned using a curated subset of the Bangla-SFT-50k dataset formatted using ChatML.

Dataset Pruning: Initial SFT experiments revealed that forcing a sub-100M parameter model to learn complex markdown syntax/tables caused severe representation crowding. Thus, the 12,517-sample Structured Formatting category was excluded. The final active training mixture consisted of 37,536 aligned pairs.
Engineering Properties: The training data strictly forbade conversational prefaces (e.g., "নিশ্চয়ই, আমি এটি করে দিচ্ছি") so that responses begin immediately with the target output, optimizing inference speeds.
Hardware: NVIDIA T4 and L4 GPUs.
Hyperparameters:
- Optimizer: Fused AdamW (adamw_torch_fused) with $β 1 = 0.9, β$

Model Architecture details

Like its base model, Alo-70M utilizes a parameter-efficient architecture:

Layers: 12
Hidden Dimension ( $d m o d e l $ ): 512 | Intermediate FFN: 1408
Attention: Grouped-Query Attention (GQA) with 8 query heads / 4 KV heads.
Positional Embeddings: RoPE (Base freq: 10,000)
Word Embeddings: Untied (tie_word_embeddings = False).
Context Window: 1024 tokens.

Evaluation Results

The model was evaluated zero-shot across Bengali reasoning and knowledge benchmarks (continuation-based log-probability evaluation):

Table with columns: Benchmark, Alo-70M (SFT), Alo-70M-Base, Gemma-3-270M-IT, TigerLLM-1B-IT
Benchmark	Alo-70M (SFT)	Alo-70M-Base	Gemma-3-270M-IT	TigerLLM-1B-IT
bangla_mmlu_bn	26.29%	26.31%	26.81%	27.66%
bangla_commonsenseqa_bn	25.88%	28.42%	22.77%	25.14%
indicbench_arc_bn_challenge	24.15%	22.70%	25.34%

Note: The 69M instruction-tuned model outperforms the larger Gemma-3-270M-IT baseline on tasks like CommonsenseQA and PIQA.

Limitations and Biases

Alignment Tax (Catastrophic Forgetting): While SFT successfully aligned the model for text generation stability and instruction following, it introduced a measurable degradation in pure zero-shot reasoning compared to the Base model (e.g., dropping from 28.42% to 25.88% on CommonsenseQA). This happens because applying instructions to a highly capacity-constrained 69M model over-indexes weights toward output formatting at the expense of some pre-trained logical representations.
Knowledge Retrieval: With under 100M parameters, the model physically lacks the capacity to serve as a comprehensive encyclopedic knowledge base. It is better suited for text processing tasks (editing, summarizing) than fact-retrieval.
Context Length: The model is optimized for a 1024-token context window. Prompts exceeding this length will be truncated or result in degraded quality.

Citation

Technical paper out soon.

Alo-70m

README

Model Summary

Related Resources

Usage

Supervised Fine-Tuning (SFT) Details

Model Architecture details

Evaluation Results

Limitations and Biases

Citation

Explore FriendliAI today

README

Model Summary

Related Resources

Usage

Supervised Fine-Tuning (SFT) Details

Model Architecture details

Evaluation Results

Limitations and Biases

Citation