stukenov

sozkz-core-llama-1b-kk-base-v1

Model Details

Table with columns: Property, Value
Property	Value
Architecture	Llama (decoder-only transformer)
Parameters	1,077.7M (~1.08B)
Hidden size	2048
Layers	22
Attention heads	16
KV heads	4 (Grouped Query Attention)
Intermediate size	5504 (SwiGLU)
Vocab size	50,257
Max sequence length	1024
Tied embeddings	Yes
Precision	bfloat16
Training features	GQA + Z-loss regularization

Training Details

Table with columns: Property, Value
Property	Value
Training tokens	~16.2B
Data mix	70% Kazakh + 30% EN-KK FineWeb-Edu
Total steps	30,899
Batch size	~524K tokens/step
Hardware	8x NVIDIA H100 SXM 80GB
Final training loss	2.636
Validation BPB	0.673
Optimizer	AdamW (cosine schedule)
License	MIT (gated access)

Training Data

70% -- stukenov/sozkz-corpus-clean-kk-text-v4: Cleaned Kazakh texts
30% -- stukenov/sozkz-corpus-clean-enkk-fineweb-edu-v2: English-Kazakh parallel data from FineWeb-Edu

Tokenizer

Custom Kazakh BPE tokenizer with 50,257 vocabulary (based on saken-tukenov/kazakh-bpe-32k, extended).

Checkpoints

Intermediate checkpoints are available as branches for research and analysis:

step-2000, step-4000, step-6000, step-8000, step-10000, step-12000, step-14000, step-16000, step-18000, step-20000, step-22000, step-24000, step-26000, step-28000, step-30000, step-final

The main branch contains the final checkpoint (step-final, step 30,899).

To load a specific checkpoint:

python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "stukenov/sozkz-core-llama-1b-kk-base-v1",
    revision="step-16000"
)

Quality Progression

Table with columns: Checkpoint, Approx. Quality, Notes
Checkpoint	Approx. Quality	Notes
step-4000	3.3 / 5	Early training, basic Kazakh grammar emerging
step-8000	3.6 / 5	Coherent sentences, some factual knowledge
step-12000	3.8 / 5	Improved fluency and factual recall
step-16000	4.0 / 5	Strong factual knowledge, good grammar
step-20000	4.1 / 5	Further refinement
step-24000

What the Model Knows

The model demonstrates knowledge of Kazakh culture, history, and geography. Example completions:

Назарбаев -- knows about the first president of Kazakhstan
Тоқаев -- recognizes the current president
Бауыржан Момышұлы -- knows the WWII hero
Абай -- knows about the great Kazakh poet and philosopher
Каспий теңізі -- factual knowledge about the Caspian Sea
Алматы, Астана -- knows major Kazakh cities

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "stukenov/sozkz-core-llama-1b-kk-base-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

prompt = "Қазақстан -- "
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

This is a base model (not instruction-tuned). It performs text completion, not Q&A.
Trained on 1024 context length only.
May generate incorrect facts (hallucination) -- do not use for factual reference.
The model was trained from scratch and may have biases present in the training data.

License

MIT License with gated access (manual approval required for models 300M+).

Citation

If you use this model, please cite:

bibtex
@misc{sozkz-llama-1b-kk-2026,
  title={SozKZ Core Llama 1B Kazakh Base v1},
  author={Saken Tukenov},
  year={2026},
  url={https://huggingface.co/stukenov/sozkz-core-llama-1b-kk-base-v1}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

stukenov

Model Tree

Base

this model

Input Modalities

Text

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Container

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Table with columns: Property, Value
Property	Value
Architecture	Llama (decoder-only transformer)
Parameters	1,077.7M (~1.08B)
Hidden size	2048
Layers	22
Attention heads	16
KV heads	4 (Grouped Query Attention)
Intermediate size	5504 (SwiGLU)
Vocab size	50,257
Max sequence length	1024
Tied embeddings	Yes
Precision	bfloat16
Training features	GQA + Z-loss regularization

Training Details

Table with columns: Property, Value
Property	Value
Training tokens	~16.2B
Data mix	70% Kazakh + 30% EN-KK FineWeb-Edu
Total steps	30,899
Batch size	~524K tokens/step
Hardware	8x NVIDIA H100 SXM 80GB
Final training loss	2.636
Validation BPB	0.673
Optimizer	AdamW (cosine schedule)
License	MIT (gated access)

Training Data

70% -- stukenov/sozkz-corpus-clean-kk-text-v4: Cleaned Kazakh texts
30% -- stukenov/sozkz-corpus-clean-enkk-fineweb-edu-v2: English-Kazakh parallel data from FineWeb-Edu

Tokenizer

Custom Kazakh BPE tokenizer with 50,257 vocabulary (based on saken-tukenov/kazakh-bpe-32k, extended).

Checkpoints

Intermediate checkpoints are available as branches for research and analysis:

The main branch contains the final checkpoint (step-final, step 30,899).

To load a specific checkpoint:

python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "stukenov/sozkz-core-llama-1b-kk-base-v1",
    revision="step-16000"
)

Quality Progression

Table with columns: Checkpoint, Approx. Quality, Notes
Checkpoint	Approx. Quality	Notes
step-4000	3.3 / 5	Early training, basic Kazakh grammar emerging
step-8000	3.6 / 5	Coherent sentences, some factual knowledge
step-12000	3.8 / 5	Improved fluency and factual recall
step-16000	4.0 / 5	Strong factual knowledge, good grammar
step-20000	4.1 / 5	Further refinement
step-24000

What the Model Knows

The model demonstrates knowledge of Kazakh culture, history, and geography. Example completions:

Назарбаев -- knows about the first president of Kazakhstan
Тоқаев -- recognizes the current president
Бауыржан Момышұлы -- knows the WWII hero
Абай -- knows about the great Kazakh poet and philosopher
Каспий теңізі -- factual knowledge about the Caspian Sea
Алматы, Астана -- knows major Kazakh cities

Usage

python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "stukenov/sozkz-core-llama-1b-kk-base-v1"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

prompt = "Қазақстан -- "
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

This is a base model (not instruction-tuned). It performs text completion, not Q&A.
Trained on 1024 context length only.
May generate incorrect facts (hallucination) -- do not use for factual reference.
The model was trained from scratch and may have biases present in the training data.

License

MIT License with gated access (manual approval required for models 300M+).

Citation

If you use this model, please cite:

bibtex
@misc{sozkz-llama-1b-kk-2026,
  title={SozKZ Core Llama 1B Kazakh Base v1},
  author={Saken Tukenov},
  year={2026},
  url={https://huggingface.co/stukenov/sozkz-core-llama-1b-kk-base-v1}
}

sozkz-core-llama-1b-kk-base-v1

README

Model Details

Training Details

Training Data

Tokenizer

Checkpoints

Quality Progression

What the Model Knows

Usage

Limitations

License

Citation

Explore FriendliAI today

README

Model Details

Training Details

Training Data

Tokenizer

Checkpoints

Quality Progression

What the Model Knows

Usage

Limitations

License

Citation