Darmm

darmm-chat-kazakh-8b

README

License: apache-2.0

Benchmarks

Zero-shot multiple-choice accuracy via option log-prob scoring (non-thinking chat template). KazMMLU: 3,000-question sample (seed 42) over all Kazakh-language subjects; Belebele: full kaz_Cyrl test (900).

Table with columns: benchmark, Qwen3-8B (base), this model
benchmark	Qwen3-8B (base)	this model
Belebele (kaz_Cyrl)	27.2%	39.8%
KazMMLU (kaz subjects)	27.9%	38.3%

Reproducible with eval_chat.py in this repo.

Usage — generation settings matter

Use sampling, not greedy decoding: like the base model, long-form Kazakh degenerates into repetition under greedy decoding. Recommended settings:

python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Darmm/darmm-chat-kazakh-8b"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

messages = [{"role": "user", "content": "Наурыз мейрамы туралы айтып бер."}]
text = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = tok(text, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs, max_new_tokens=512,
    do_sample=True, temperature=0.7, top_p=0.8, top_k=20, repetition_penalty=1.1,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The model is trained for non-thinking mode (enable_thinking=False).

Training

QLoRA: 4-bit NF4 base, LoRA r=32 α=64 on all attention+MLP projections (59.9M trainable params)
Data: AmanMussa/kazakh-instruction-v2 (51k) + sabinaasker/kazakh_dolly (15k) + Russian (d0rj/alpaca-cleaned-ru, 25k) + English (yahma/alpaca-cleaned, 15k)
1 epoch, effective batch 32, lr 1e-4 cosine, max_seq 1024, bf16 compute, single A100

The LoRA adapter alone is at Darmm/darmm-chat-kazakh-8b-lora.

Limitations

Factual reliability is limited (38% KazMMLU): it can state dates, names, and facts confidently but wrongly — do not use as a factual reference without verification/RAG.
Much of the Kazakh training data is machine-translated; phrasing can be unnatural and occasionally awkward.
Greedy decoding degrades output — always use the sampling settings above.
Not safety-tuned beyond what the base model provides.

Built by Darmm · 2026-09 · Apache 2.0

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Darmm

Model Tree

Base

Qwen/Qwen3-8B

Fine-tuned

this model

Input Modalities

Text

Output Modalities