MiniCPM5-1B-Pashto API & Inference Endpoint

Pashto + Urdu Tokenizer Surgery Model

📋 Model Overview

This is a tokenizer-modified version of openbmb/MiniCPM5-1B with 46 Pashto and Urdu-specific characters added as single tokens to the vocabulary.

Table with columns: Property, Value
Property	Value
Base Model	`openbmb/MiniCPM5-1B`
Original Vocab Size	130,560
New Vocab Size	130,606
Atoms Added	46
Architecture	LlamaForCausalLM
Hidden Size	1,536
Layers	24
Context Length	131,072
Embeddings	NOT tied (independent input/output)

🔍 Why This Model Exists

MiniCPM5-1B's tokenizer does not natively support many Pashto and Urdu characters as single tokens. This causes:

❌ Characters being split into multiple tokens
❌ Inefficient encoding
❌ Poor representation for Pashto/Urdu text

This model fixes that by surgically adding 46 missing atoms as independent tokens.

📊 Complete Audit & Modification Report

✅ Existing Single-Token Atoms (20)

Table with columns: Atom, ID, Atom, ID
Atom	ID	Atom	ID
`ا`	20541	`د`	57692
`ب`	79922	`ر`	37101
`پ`	120234	`ز`

⚠️ Split Atoms — FIXED via Surgery (46)

Table with columns: #, Atom, Old IDs, New ID
#	Atom	Old IDs	New ID
1	`ښ`	[172, 270]	130560
2	`څ`	[172, 249]	130561
3	`ځ`	[172, 245]	130562
4

🔬 Embedding Forensics

Model Configuration

python
{
    "architectures": ["LlamaForCausalLM"],
    "vocab_size": 130606,
    "hidden_size": 1536,
    "num_hidden_layers": 24,
    "max_position_embeddings": 131072,
    "tie_word_embeddings": False,
    "initializer_range": 0.02,
}

Input Embedding Statistics (New Atoms)

Table with columns: Metric, Value
Metric	Value
Mean embedding norm	0.901353
Max cosine similarity	0.25996944

Most Similar Pair (New Atoms)

Table with columns: Atom 1, Atom 2, Cosine Similarity
Atom 1	Atom 2	Cosine Similarity
`ح`	`ع`	0.25996944

✅ No exact duplicate embeddings detected

Sample New Token Embedding Fingerprints

Table with columns: Atom, ID, Input Norm, Input Mean, Input Std
Atom	ID	Input Norm	Input Mean	Input Std
`ښ`	130560	0.780407	0.000240	0.019918
`څ`	130561	0.789077	-0.000997	0.020116
`ځ`	130562

🚀 Usage

Installation

bash
pip install transformers torch accelerate

Basic Loading

python
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model and tokenizer
model_id = "nassimjp/MiniCPM5-1B-Pashto"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Tokenization Check

python
# Test Pashto characters
pashto_text = "سلام ښه راغلئ"

# Tokenize
tokens = tokenizer.encode(pashto_text, add_special_tokens=False)
print(f"Token IDs: {tokens}")

# Decode back
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")

# Check individual atoms
for char in "ښڅځڼږډټړېۍګ":
    ids = tokenizer.encode(char, add_special_tokens=False)
    print(f"'{char}' → {ids}")

Text Generation

python
import torch

# Simple generation
prompt = "ښه راغلئ"
inputs = tokenizer.encode(prompt, return_tensors="pt")

# Generate
with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=50,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Batch Processing

python
texts = [
    "سلام",
    "ښه راغلئ",
    "پښتو ژبه",
    "ستاسو نوم څه دی؟"
]

# Tokenize batch
encodings = tokenizer(
    texts,
    padding=True,
    return_tensors="pt"
)

print(encodings.input_ids.shape)  # (4, sequence_length)

⚙️ Technical Details

Surgery Methodology

Audit Phase
- Iterated through 66 Pashto/Urdu characters
- Detected which characters split into multiple tokens
- Identified 46 characters needing surgery
Token Addition
- Used tokenizer.add_tokens() to add 46 missing atoms
- Resized model embeddings with model.resize_token_embeddings()
- New tokens assigned IDs: 130560 → 130605
Independent Initialization
- Input embeddings initialized with normal distribution
- Output embeddings initialized independently

⚠️ Important Notes

Table with columns: Note, Description
Note	Description
No Finetuning	This model has NOT been finetuned on Pashto/Urdu data
Embeddings	Input and output embeddings are NOT tied
New Tokens	Added tokens are at the end of vocabulary (IDs 130560+)
Performance	Requires finetuning on Pashto/Urdu data for optimal performance
Tokenizer	Works out-of-the-box with all Pashto/Urdu atoms

📦 Model Files

markdown
MiniCPM5-1B-Pashto/
├── chat_template.jinja             0.01 MB
├── config.json                     0.00 MB
├── generation_config.json          0.00 MB
├── ipashto_surgery_info.txt        0.00 MB
├── model-00001-of-00002.safetensors  1,899.42 MB
├── model-00002-of-00002.safetensors    162.02 MB
├── model.safetensors.index.json    0.02 MB
├── tokenizer.json                  9.44 MB
└── tokenizer_config.json           0.00 MB

Total Size: 2.02 GB

Base Model: openbmb/MiniCPM5-1B
Original Paper: MiniCPM: Unveiling the Potential of Small Language Models
GitHub: OpenBMB/MiniCPM

📝 Citation

If you use this model, please cite the original MiniCPM paper:

bibtex
@article{minicpm2024,
  title={MiniCPM: Unveiling the Potential of Small Language Models},
  author={Hu, Shengding and Ding, Ning and others},
  journal={arXiv preprint arXiv:2404.06395},
  year={2024}
}

📄 License

This model is released under the Apache License 2.0.

🤝 Acknowledgements

OpenBMB for developing and releasing MiniCPM5-1B
Hugging Face for the transformers library and model hosting
Kaggle for providing the compute environment

🧪 Testing Results

All 66 Pashto/Urdu atoms successfully tokenize as single tokens:

Table with columns: Category, Count, Status
Category	Count	Status
Pashto-Specific	11	✅ All single tokens
Shared Arabic/Persian	30	✅ All single tokens
Positional Forms	3	✅ All single tokens
Urdu/South Asian	6	✅ All single tokens
Arabic Variants	6	✅ All single tokens
Eastern Arabic Digits	10

📞 Contact & Support

Model Page: nassimjp/MiniCPM5-1B-Pashto
Issues: Please open an issue on the model page
Suggestions: Feedback welcome!

Created with ❤️ for the Pashto and Urdu communities

markdown
---

Pashto + Urdu Tokenizer Surgery Model

📋 Model Overview

This is a tokenizer-modified version of openbmb/MiniCPM5-1B with 46 Pashto and Urdu-specific characters added as single tokens to the vocabulary.

Table with columns: Property, Value
Property	Value
Base Model	`openbmb/MiniCPM5-1B`
Original Vocab Size	130,560
New Vocab Size	130,606
Atoms Added	46
Architecture	LlamaForCausalLM
Hidden Size	1,536
Layers	24
Context Length	131,072
Embeddings	NOT tied (independent input/output)

🔍 Why This Model Exists

MiniCPM5-1B's tokenizer does not natively support many Pashto and Urdu characters as single tokens. This causes:

❌ Characters being split into multiple tokens
❌ Inefficient encoding
❌ Poor representation for Pashto/Urdu text

This model fixes that by surgically adding 46 missing atoms as independent tokens.

📊 Complete Audit & Modification Report

✅ Existing Single-Token Atoms (20)

Table with columns: Atom, ID, Atom, ID
Atom	ID	Atom	ID
`ا`	20541	`د`	57692
`ب`	79922	`ر`	37101
`پ`	120234	`ز`

⚠️ Split Atoms — FIXED via Surgery (46)

Table with columns: #, Atom, Old IDs, New ID
#	Atom	Old IDs	New ID
1	`ښ`	[172, 270]	130560
2	`څ`	[172, 249]	130561
3	`ځ`	[172, 245]	130562
4

🔬 Embedding Forensics

Model Configuration

python
{
    "architectures": ["LlamaForCausalLM"],
    "vocab_size": 130606,
    "hidden_size": 1536,
    "num_hidden_layers": 24,
    "max_position_embeddings": 131072,
    "tie_word_embeddings": False,
    "initializer_range": 0.02,
}

Input Embedding Statistics (New Atoms)

Table with columns: Metric, Value
Metric	Value
Mean embedding norm	0.901353
Max cosine similarity	0.25996944

Most Similar Pair (New Atoms)

Table with columns: Atom 1, Atom 2, Cosine Similarity
Atom 1	Atom 2	Cosine Similarity
`ح`	`ع`	0.25996944

✅ No exact duplicate embeddings detected

Sample New Token Embedding Fingerprints

Table with columns: Atom, ID, Input Norm, Input Mean, Input Std
Atom	ID	Input Norm	Input Mean	Input Std
`ښ`	130560	0.780407	0.000240	0.019918
`څ`	130561	0.789077	-0.000997	0.020116
`ځ`	130562

🚀 Usage

Installation

bash
pip install transformers torch accelerate

Basic Loading

python
from transformers import AutoTokenizer, AutoModelForCausalLM

# Load model and tokenizer
model_id = "nassimjp/MiniCPM5-1B-Pashto"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Tokenization Check

python
# Test Pashto characters
pashto_text = "سلام ښه راغلئ"

# Tokenize
tokens = tokenizer.encode(pashto_text, add_special_tokens=False)
print(f"Token IDs: {tokens}")

# Decode back
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")

# Check individual atoms
for char in "ښڅځڼږډټړېۍګ":
    ids = tokenizer.encode(char, add_special_tokens=False)
    print(f"'{char}' → {ids}")

Text Generation

python
import torch

# Simple generation
prompt = "ښه راغلئ"
inputs = tokenizer.encode(prompt, return_tensors="pt")

# Generate
with torch.no_grad():
    outputs = model.generate(
        inputs,
        max_new_tokens=50,
        temperature=0.7,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Batch Processing

python
texts = [
    "سلام",
    "ښه راغلئ",
    "پښتو ژبه",
    "ستاسو نوم څه دی؟"
]

# Tokenize batch
encodings = tokenizer(
    texts,
    padding=True,
    return_tensors="pt"
)

print(encodings.input_ids.shape)  # (4, sequence_length)

⚙️ Technical Details

Surgery Methodology

Audit Phase
- Iterated through 66 Pashto/Urdu characters
- Detected which characters split into multiple tokens
- Identified 46 characters needing surgery
Token Addition
- Used tokenizer.add_tokens() to add 46 missing atoms
- Resized model embeddings with model.resize_token_embeddings()
- New tokens assigned IDs: 130560 → 130605
Independent Initialization
- Input embeddings initialized with normal distribution
- Output embeddings initialized independently

⚠️ Important Notes

Table with columns: Note, Description
Note	Description
No Finetuning	This model has NOT been finetuned on Pashto/Urdu data
Embeddings	Input and output embeddings are NOT tied
New Tokens	Added tokens are at the end of vocabulary (IDs 130560+)
Performance	Requires finetuning on Pashto/Urdu data for optimal performance
Tokenizer	Works out-of-the-box with all Pashto/Urdu atoms

📦 Model Files

markdown
MiniCPM5-1B-Pashto/
├── chat_template.jinja             0.01 MB
├── config.json                     0.00 MB
├── generation_config.json          0.00 MB
├── ipashto_surgery_info.txt        0.00 MB
├── model-00001-of-00002.safetensors  1,899.42 MB
├── model-00002-of-00002.safetensors    162.02 MB
├── model.safetensors.index.json    0.02 MB
├── tokenizer.json                  9.44 MB
└── tokenizer_config.json           0.00 MB

Total Size: 2.02 GB

Base Model: openbmb/MiniCPM5-1B
Original Paper: MiniCPM: Unveiling the Potential of Small Language Models
GitHub: OpenBMB/MiniCPM

📝 Citation

If you use this model, please cite the original MiniCPM paper:

bibtex
@article{minicpm2024,
  title={MiniCPM: Unveiling the Potential of Small Language Models},
  author={Hu, Shengding and Ding, Ning and others},
  journal={arXiv preprint arXiv:2404.06395},
  year={2024}
}

📄 License

This model is released under the Apache License 2.0.

🤝 Acknowledgements

OpenBMB for developing and releasing MiniCPM5-1B
Hugging Face for the transformers library and model hosting
Kaggle for providing the compute environment

🧪 Testing Results

All 66 Pashto/Urdu atoms successfully tokenize as single tokens:

Table with columns: Category, Count, Status
Category	Count	Status
Pashto-Specific	11	✅ All single tokens
Shared Arabic/Persian	30	✅ All single tokens
Positional Forms	3	✅ All single tokens
Urdu/South Asian	6	✅ All single tokens
Arabic Variants	6	✅ All single tokens
Eastern Arabic Digits	10

📞 Contact & Support

Model Page: nassimjp/MiniCPM5-1B-Pashto
Issues: Please open an issue on the model page
Suggestions: Feedback welcome!

Created with ❤️ for the Pashto and Urdu communities

markdown
---

MiniCPM5-1B-Pashto

README

Pashto + Urdu Tokenizer Surgery Model

📋 Model Overview

🔍 Why This Model Exists

📊 Complete Audit & Modification Report

✅ Existing Single-Token Atoms (20)

⚠️ Split Atoms — FIXED via Surgery (46)

🔬 Embedding Forensics

Model Configuration

Input Embedding Statistics (New Atoms)

Most Similar Pair (New Atoms)

Sample New Token Embedding Fingerprints

🚀 Usage

Installation

Basic Loading

Tokenization Check

Text Generation

Batch Processing

⚙️ Technical Details

Surgery Methodology

⚠️ Important Notes

📦 Model Files

🔗 Related Resources

📝 Citation

📄 License

🤝 Acknowledgements

🧪 Testing Results

📞 Contact & Support

Explore FriendliAI today

README

Pashto + Urdu Tokenizer Surgery Model

📋 Model Overview

🔍 Why This Model Exists

📊 Complete Audit & Modification Report

✅ Existing Single-Token Atoms (20)

⚠️ Split Atoms — FIXED via Surgery (46)

🔬 Embedding Forensics

Model Configuration

Input Embedding Statistics (New Atoms)

Most Similar Pair (New Atoms)

Sample New Token Embedding Fingerprints

🚀 Usage

Installation

Basic Loading

Tokenization Check

Text Generation

Batch Processing

⚙️ Technical Details

Surgery Methodology

⚠️ Important Notes

📦 Model Files

🔗 Related Resources

📝 Citation

📄 License

🤝 Acknowledgements

🧪 Testing Results

📞 Contact & Support