Evaluation
Vezilka FINKI Lecture ASR Benchmark (the finki-benchmark configuration of
vezilka-ai/macedonian-speech) - five held-out FINKI
computer-science lectures with manually reviewed references; neither their audio nor
their transcripts were used in training. Normalised WER/CER in % (NFC, lower-cased,
punctuation removed); CUDA, float32, batch 1, beam 4.
Table with columns: view, clips, WER, CER, S, I, D| view | clips | WER | CER | S | I | D |
|---|
| full | 1,406 | 4.51 | 1.52 | 1,712 | 298 | 425 |
| speaker_balanced | 892 | 4.15 | 1.36 | 1,012 | 169 | 239 |
Usage
from peft import PeftModelfrom transformers import WhisperForConditionalGeneration, WhisperProcessor processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3-turbo")base = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3-turbo")model = PeftModel.from_pretrained(base, "vezilka-ai/vezilka-whisper-large-v3-turbo-mk-lora").merge_and_unload() features = processor(audio_16khz, sampling_rate=16000, return_tensors="pt").input_featuresids = model.generate(features, language="macedonian", task="transcribe", num_beams=4)print(processor.batch_decode(ids, skip_special_tokens=True)[0])
Training data
Continued from an earlier project adapter (r=64, q/k/v/o, cosine schedule, LR 5e-4) for 5 epochs at peak LR 5e-4 on accepted Doniraj donations (submissions-20260831), the project's prvatv_dataset, and a 15-hour replay subset of videa_so_transkript_od_yt.
License
Apache-2.0. The base model openai/whisper-large-v3-turbo is
released under the MIT license.
Acknowledgements
Supported by the Ministry of Education and Science of the Republic of North Macedonia (project "Utilizing AI and National Large Language Models to Advance Macedonian Language Capabilities") and co-funded by the EC/EuroHPC JU, the Ministry of Digital Transformation of the Republic of North Macedonia and the Faculty of Computer Science and Engineering (FINKI), Ss. Cyril and Methodius University in Skopje, through the VEZILKA project, Grant Agreement No. 101263128.
Citation
The data article describing this collection (Gajdov, Karamachoski, Stojcheva, Mishkovski and Mishev, Curated Macedonian speech datasets with regional transcription layers and a manually reviewed FINKI lecture benchmark) is under review; its citation will be added here on publication.