AhiskaAI

AhiskaAI-134m-Base-v0.2

Deploy Dedicated

README

License: apache-2.0

Model Details

Architecture: Llama-based architecture.
Parameters: 134M.
Context Window: 1024 tokens.
Tokenizer: Custom BPE Tokenizer (Vocabulary Size: 32,000).
Training Framework: PyTorch & Transformers.

Data Curation (The "Quality over Quantity" Approach)

A major leap in the v0.2 release is the transition to data-centric AI.

Raw Data (v0.1): 5GB of raw corpus.
Curated Data (v0.2): 1.2GB of meticulously filtered, high-quality Turkish text.
Process: Approximately 75% of low-quality, noisy, and redundant data was eliminated to ensure the model focuses on grammatical correctness and linguistic density.

Key Improvements from v0.1

Architecture Shift: Migrated from standard GPT-2 to a modern Llama-based architecture for superior sequence modeling.
Normalization: Implemented RMSNorm for training stability.
Positional Encoding: Adopted RoPE (Rotary Positional Embeddings).
Activation: Switched to SiLU activation function.
Precision: Trained in bfloat16 for efficiency on consumer-grade hardware.

Training Logs

Training Loss Curve The graph above demonstrates the training convergence of AhiskaAI-134m-Base-v0.2. The stable decline in loss confirms the effective alignment of the model architecture with the curated Turkish dataset.

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("AhiskaAI/AhiskaAI-134m-Base-v0.2")
tokenizer = AutoTokenizer.from_pretrained("AhiskaAI/AhiskaAI-134m-Base-v0.2")

text = "Türkiye Cumhuriyeti"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Hardware

Trained on NVIDIA RTX 4050 6GB Laptop GPU.

Future Plans

Instruct Version: AhiskaAI-134m-IT-v0.2 (Fine-tuned for chat and instructions).

Preference Alignment: v0.2.1 DPO (Direct Preference Optimization) is currently under development to refine response quality.

About AhiskaAI

AhiskaAI is an independent initiative dedicated to pushing the boundaries of Small Language Models in the Turkish language. Follow us on Hugging Face for updates and new releases.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

AhiskaAI

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Container

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Details

Architecture: Llama-based architecture.
Parameters: 134M.
Context Window: 1024 tokens.
Tokenizer: Custom BPE Tokenizer (Vocabulary Size: 32,000).
Training Framework: PyTorch & Transformers.

Data Curation (The "Quality over Quantity" Approach)

A major leap in the v0.2 release is the transition to data-centric AI.

Raw Data (v0.1): 5GB of raw corpus.
Curated Data (v0.2): 1.2GB of meticulously filtered, high-quality Turkish text.
Process: Approximately 75% of low-quality, noisy, and redundant data was eliminated to ensure the model focuses on grammatical correctness and linguistic density.

Key Improvements from v0.1

Architecture Shift: Migrated from standard GPT-2 to a modern Llama-based architecture for superior sequence modeling.
Normalization: Implemented RMSNorm for training stability.
Positional Encoding: Adopted RoPE (Rotary Positional Embeddings).
Activation: Switched to SiLU activation function.
Precision: Trained in bfloat16 for efficiency on consumer-grade hardware.

Training Logs

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("AhiskaAI/AhiskaAI-134m-Base-v0.2")
tokenizer = AutoTokenizer.from_pretrained("AhiskaAI/AhiskaAI-134m-Base-v0.2")

text = "Türkiye Cumhuriyeti"
inputs = tokenizer(text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Hardware

Trained on NVIDIA RTX 4050 6GB Laptop GPU.

Future Plans

Instruct Version: AhiskaAI-134m-IT-v0.2 (Fine-tuned for chat and instructions).

Preference Alignment: v0.2.1 DPO (Direct Preference Optimization) is currently under development to refine response quality.

About AhiskaAI

AhiskaAI is an independent initiative dedicated to pushing the boundaries of Small Language Models in the Turkish language. Follow us on Hugging Face for updates and new releases.