Model Details
Table with columns: Field, Value| Field | Value |
|---|
| Base model | openai/whisper-large-v3 |
| Adaptation method | LoRA / PEFT |
| Model type | Whisper encoder-decoder ASR model |
| Task | Automatic Speech Recognition / Speech-to-Text |
| Languages | Kazakh, Russian, Kazakh-Russian mixed speech |
| Repository type | LoRA adapter |
| Project context | Academic thesis research |
LoRA Configuration
Table with columns: Parameter, Value| Parameter | Value |
|---|
Rank r | 32 |
Alpha α | 64 |
| Dropout | 0.05 |
| Target modules | q_proj, v_proj, k_proj, out_proj |
Recommended Use
This model is mainly intended for:
- academic ASR research;
- Kazakh speech recognition experiments;
- Kazakh-Russian mixed-speech transcription;
- code-switching ASR evaluation;
- comparison with other KRASR ASR models;
- reproducibility of thesis experiments;
- demonstration in speech-to-text applications.
This is the stronger Whisper-based model in the KRASR comparison. It usually requires more compute than Whisper-Small, but gives much better recognition quality.
Quick Start
Install the required libraries:
pip install -U transformers peft accelerate torch librosa soundfile evaluate tqdm
Run inference on one audio file
import torch
import librosa
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor, pipeline
base_model_id = "openai/whisper-large-v3"
adapter_id = "KRASR/kazakh-russian-asr-whisper-large-v3-lora"
audio_path = "audio.wav"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
pipeline_device = 0 if torch.cuda.is_available() else -1
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
processor = WhisperProcessor.from_pretrained(base_model_id)
base_model = WhisperForConditionalGeneration.from_pretrained(
base_model_id,
torch_dtype=torch_dtype,
low_cpu_mem_usage=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model = model.merge_and_unload()
model.to(device)
model.eval()
asr = pipeline(
task="automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
torch_dtype=torch_dtype,
device=pipeline_device,
)
audio, sr = librosa.load(audio_path, sr=16000, mono=True)
result = asr(
{"array": audio, "sampling_rate": sr},
generate_kwargs={
"task": "transcribe",
"language": "kazakh",
"num_beams": 3,
"no_repeat_ngram_size": 4,
"repetition_penalty": 1.12,
},
)
print(result["text"])
Decoding Notes
For normal testing, avoid using a fixed max_new_tokens value such as 96.
A fixed limit can accidentally cut off longer transcriptions.
A good starting point for Kazakh-dominant mixed speech is:
generate_kwargs = {
"task": "transcribe",
"language": "kazakh",
"num_beams": 3,
"no_repeat_ngram_size": 4,
"repetition_penalty": 1.12,
}
Why forced Kazakh decoding?
In mixed Kazakh-Russian speech, automatic language detection can be unstable, especially on short utterances.
If the audio is mostly Kazakh with Russian insertions, forcing Kazakh decoding usually keeps the transcription closer to the target speech domain.
Russian words may still appear in the output when the model recognizes them from the audio.
Optional dynamic token limit for apps or batch evaluation
For applications or controlled batch evaluation, a dynamic limit is safer than one fixed value.
The following helper follows the same idea as the KRASR demo module: short clips receive a smaller output limit, while longer clips receive a larger limit.
def build_generate_kwargs(
audio_duration_sec: float | None,
language: str | None = "kazakh",
num_beams: int = 3,
) -> dict:
if audio_duration_sec is None:
max_new_tokens = 96
elif audio_duration_sec <= 5:
max_new_tokens = 40
elif audio_duration_sec <= 10:
max_new_tokens = 64
elif audio_duration_sec <= 15:
max_new_tokens = 80
elif audio_duration_sec <= 20:
max_new_tokens = 96
elif audio_duration_sec <= 25:
max_new_tokens = 112
else:
max_new_tokens = 128
generate_kwargs = {
"task": "transcribe",
"num_beams": int(num_beams),
"max_new_tokens": max_new_tokens,
"no_repeat_ngram_size": 4,
"repetition_penalty": 1.12,
}
if language is not None:
generate_kwargs["language"] = language
return generate_kwargs
Use this only when you specifically need output-length control. For ordinary one-file testing, starting without max_new_tokens is usually simpler.
Hardware Notes
Whisper Large-v3 is significantly heavier than Whisper-Small-based models.
Recommended setup:
- GPU inference is strongly recommended;
- use
torch.float16 on CUDA when possible;
- CPU inference may be slow;
- memory requirements are higher than for Whisper Small LoRA.
If you need a lighter model, use KRASR/kazakh-russian-asr-whisper-small-lora or another smaller KRASR model.
Evaluation Results
Table with columns: Evaluation set, WER, CER, Notes| Evaluation set | WER | CER | Notes |
|---|
| Test-MIXED | 0.3049 | 0.1496 | Main Kazakh-Russian mixed-speech test set |
| Test-KK | 0.1730 | 0.0392 | Internal pure Kazakh test set |
| Test-RU | 0.0682 | 0.0144 | Internal pure Russian test set |
| FLEURS-KK | 0.2053 | 0.0700 | External Kazakh benchmark |
This model achieved the best word-level result among the evaluated KRASR systems. In the thesis evaluation, Whisper Large-v3 LoRA was used as a stronger comparative model rather than the main lightweight deployment model.
For ordinary testing, a fixed max_new_tokens value is not required. In the thesis comparison, Whisper Large-v3 LoRA was evaluated with a stable beam-search configuration.
Batch Evaluation Example
The following example shows how to test the model on a simple JSONL manifest.
Expected manifest format:
{"audio": "path/to/audio.wav", "text": "reference transcription"}
Evaluation script:
import json
import torch
import librosa
import evaluate
from tqdm import tqdm
from peft import PeftModel
from transformers import WhisperForConditionalGeneration, WhisperProcessor, pipeline
base_model_id = "openai/whisper-large-v3"
adapter_id = "KRASR/kazakh-russian-asr-whisper-large-v3-lora"
manifest_path = "test_manifest.jsonl"
device = "cuda:0" if torch.cuda.is_available() else "cpu"
pipeline_device = 0 if torch.cuda.is_available() else -1
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
wer_metric = evaluate.load("wer")
cer_metric = evaluate.load("cer")
def build_generate_kwargs(audio_duration_sec, language="kazakh", num_beams=3):
if audio_duration_sec is None:
max_new_tokens = 96
elif audio_duration_sec <= 5:
max_new_tokens = 40
elif audio_duration_sec <= 10:
max_new_tokens = 64
elif audio_duration_sec <= 15:
max_new_tokens = 80
elif audio_duration_sec <= 20:
max_new_tokens = 96
elif audio_duration_sec <= 25:
max_new_tokens = 112
else:
max_new_tokens = 128
generate_kwargs = {
"task": "transcribe",
"num_beams": int(num_beams),
"max_new_tokens": max_new_tokens,
"no_repeat_ngram_size": 4,
"repetition_penalty": 1.12,
}
if language is not None:
generate_kwargs["language"] = language
return generate_kwargs
processor = WhisperProcessor.from_pretrained(base_model_id)
base_model = WhisperForConditionalGeneration.from_pretrained(
base_model_id,
torch_dtype=torch_dtype,
low_cpu_mem_usage=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model = model.merge_and_unload()
model.to(device)
model.eval()
asr = pipeline(
task="automatic-speech-recognition",
model=model,
tokenizer=processor.tokenizer,
feature_extractor=processor.feature_extractor,
torch_dtype=torch_dtype,
device=pipeline_device,
)
predictions = []
references = []
with open(manifest_path, "r", encoding="utf-8") as f:
rows = [json.loads(line) for line in f]
for row in tqdm(rows):
audio, sr = librosa.load(row["audio"], sr=16000, mono=True)
duration_sec = len(audio) / sr
result = asr(
{"array": audio, "sampling_rate": sr},
generate_kwargs=build_generate_kwargs(
audio_duration_sec=duration_sec,
language="kazakh",
num_beams=3,
),
)
predictions.append(result["text"])
references.append(row["text"])
wer = wer_metric.compute(predictions=predictions, references=references)
cer = cer_metric.compute(predictions=predictions, references=references)
print(f"WER: {wer:.4f}")
print(f"CER: {cer:.4f}")
Training Data
The model was fine-tuned using the KRASR/kazakh-russian-asr-dataset, prepared for Kazakh and Kazakh-Russian mixed-speech ASR experiments.
The dataset preparation workflow included:
- source selection;
- audio segmentation;
- transcription review;
- text normalization;
- train/validation/test split preparation;
- evaluation setup for mixed-language ASR.
The dataset was prepared for speech recognition only. It was not designed for speaker identification, biometric analysis, or demographic classification.
Preprocessing
Audio and text were prepared using a consistent ASR preprocessing pipeline.
Audio preprocessing:
- mono audio;
- 16 kHz sampling rate;
- short-segment ASR setting.
Text normalization included:
- lowercasing;
- whitespace normalization;
- punctuation cleanup;
- preservation of Kazakh-specific letters;
- preservation of Russian words in mixed utterances;
- removal of formatting noise that does not affect transcription meaning.
Known Limitations
The model may make errors on:
- very short audio clips;
- noisy recordings;
- overlapping speech;
- informal conversational speech;
- rare names, places, and domain-specific terms;
- long Russian segments inside Kazakh-dominant speech;
- silent or low-quality audio.
Like other Whisper-based models, it may sometimes produce extra words or hallucinated text, especially when the input audio is too short, unclear, or contains long silence.
Out-of-Scope Use
This model is not intended for:
- speaker identification;
- biometric profiling;
- demographic classification;
- surveillance or tracking of individuals;
- high-stakes decision-making systems;
- production deployment without additional validation.
Project Context
KRASR was created as part of an academic thesis project on automatic Kazakh speech-to-text conversion using fine-tuned multilingual ASR models.
The project compares Whisper-Small baseline, Whisper-Small LoRA, Whisper-Small full fine-tuning, XLS-R 1B CTC, and Whisper Large-v3 LoRA on Kazakh, Russian, and Kazakh-Russian mixed speech.
KRASR/kazakh-russian-asr-datasetKRASR/kazakh-russian-asr-whisper-small-loraKRASR/kazakh-russian-asr-whisper-small-full-ftKRASR/kazakh-russian-asr-whisper-large-v3-loraKRASR/kazakh-russian-asr-xls-r-1b-ctcKRASR/kazakh-russian-speech-to-text-module
Citation
There is no formal publication for this model yet.
If you use this model or dataset in academic work, please cite or mention the KRASR Hugging Face repository and the related thesis project:
@misc{krasr_whisper_large_v3_lora_2026,
title = {Kazakh-Russian ASR Whisper Large-v3 LoRA},
author = {Mukhambet, Madiyar and Makhmud, Danial},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/KRASR/kazakh-russian-asr-whisper-large-v3-lora}},
note = {LoRA adapter for Kazakh and Kazakh-Russian mixed-speech ASR}
}
Related thesis project:
Madiyar Mukhambet and Danial Makhmud.
Development of a Software Module for Automatic Kazakh Speech-to-Text Conversion Based on Fine-Tuned Whisper-Small Model.
Astana IT University, 2026.