GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
waxal-benchmarking
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
waxal-benchmarking
Model Tree
openai/whisper-small
Input Modalities
Output Modalities
Supported Functionality
| Setting | WER | CER |
|---|---|---|
| Leave-Luganda-out (18 other languages, same recipe) | 86.4 | 32.0 |
| This model (Luganda only, same recipe) | 283.0 | 158.0 |
| Released per-language Whisper-Small (benchmark recipe) | 21.6 | n/a |
Test utterances scored: 638.
Known failure: language drift. Luganda is not a Whisper language and this recipe decodes without a language token, so Whisper's language detection often falls back to English. With only 5,455 Luganda training utterances, fine-tuning did not override that prior: 124 of 638 test utterances become English, repetitive output more than 3x the reference length, which drives WER above 100 (112.6 WER on the remaining utterances). Retraining for 1,400 steps (epoch-matched to the Ewe and Fula runs) made this worse (192/638 runaway, 373.6 WER), so it is not overfitting. The 18-language leave-one-out model sees about 54k utterances without a language token and does not show this failure.
All Luganda utterances in the WAXAL train split
(google/WaxalNLP), 16 kHz mono;
transcripts NFC-normalized, lower-cased, punctuation removed, diacritics preserved.
Labels longer than Whisper's 448-token limit are truncated.
| Hyperparameter | Value |
|---|---|
| Base | openai/whisper-small (244M) |
| Training languages | 1 (Luganda) |
| Steps | 4,000 |
| Optimizer | AdamW, lr 1e-5, 200 warmup steps |
| Batch size | 16 |
| Precision | fp16 |
| Language token | none (same as the leave-one-out model) |
| Hardware | 1x NVIDIA H200 |
Greedy decoding on the full Luganda WAXAL test split; WER and CER with jiwer on
NFC-normalized, lower-cased text with diacritics preserved.
python
from transformers import WhisperForConditionalGeneration, WhisperProcessorimport soundfile as sf model = WhisperForConditionalGeneration.from_pretrained("waxal-benchmarking/whisper-small-waxal-matched-lug")processor = WhisperProcessor.from_pretrained("waxal-benchmarking/whisper-small-waxal-matched-lug")wav, sr = sf.read("your_audio.wav") # 16 kHz monofeats = processor(wav, sampling_rate=16000, return_tensors="pt").input_featuresids = model.generate(feats)print(processor.batch_decode(ids, skip_special_tokens=True))Part of the WAXAL ASR Benchmark (arXiv:2606.02375).
bibtex
@article{waxalnet2026, title = {The WAXAL ASR Benchmark: Fine-Tuned Edge Models Across 19 African Languages}, author = {Olufemi, Victor Tolulope and Babatunde, Oreoluwa and Njema, Ramsey and others}, year = {2026}, note = {arXiv preprint arXiv:2606.02375}}Supported by Lynguallabs, Open Token, and CMU Africa.
from transformers import WhisperForConditionalGeneration, WhisperProcessorimport soundfile as sf model = WhisperForConditionalGeneration.from_pretrained("waxal-benchmarking/whisper-small-waxal-matched-lug")processor = WhisperProcessor.from_pretrained("waxal-benchmarking/whisper-small-waxal-matched-lug")wav, sr = sf.read("your_audio.wav") # 16 kHz monofeats = processor(wav, sampling_rate=16000, return_tensors="pt").input_featuresids = model.generate(feats)print(processor.batch_decode(ids, skip_special_tokens=True))@article{waxalnet2026, title = {The WAXAL ASR Benchmark: Fine-Tuned Edge Models Across 19 African Languages}, author = {Olufemi, Victor Tolulope and Babatunde, Oreoluwa and Njema, Ramsey and others}, year = {2026}, note = {arXiv preprint arXiv:2606.02375}}