GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
waxal-benchmarking
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
waxal-benchmarking
Model Tree
openai/whisper-small
Input Modalities
Output Modalities
Supported Functionality
WAXAL's headline finding is that in-language data is essential. Whisper (unlike Omnilingual) does not cover most WAXAL languages in pretraining, so it lets us ask a clean question with no forgetting confound: can training on sibling languages build a language the model never saw? Here Luganda has Bantu relatives (incl. Lingala and Shona) that Whisper itself was pretrained on.
| Setting | WER | CER |
|---|---|---|
| Whisper zero-shot (base) | Unsupported (not in Whisper's 99 languages) | — |
| This model (trained on 18 siblings, Luganda held out) | 86.4 | 32.0 |
| Monolingual Whisper-Small (trained on Luganda) | 21.6 | — |
The gap is stark: sibling training lands far above the monolingual model — training on neighbours cannot substitute for in-language data, even for a genuinely unseen language.
Whisper-Small fine-tuned on the pooled train splits of the 18 non-Luganda WAXAL
languages (google/WaxalNLP),
balanced by capping each language, multilingual (no forced language token). 16 kHz mono;
transcripts NFC-normalized + lower-cased, punctuation removed, diacritics preserved.
| Hyperparameter | Value |
|---|---|
| Base | openai/whisper-small (244M) |
| Training languages | 18 (all WAXAL except Luganda) |
| Utterances / language | 3,000 (balanced) |
| Steps | 4,000 |
| Optimizer | AdamW, lr 1e-5, 200 warmup |
| Batch size | 16 |
| Precision | fp16 |
| Hardware | 1× H200 |
Languages it was trained on: Acholi, Akan, Amharic, Dagbani, Dagaare, Ewe, Fula, Ikposo, Lingala, Masaaba, Malagasy, Nyankole, Oromo, Sidama, Shona, Soga, Tigrinya, Wolaytta.
Scored on the held-out Luganda test split (jiwer WER/CER, NFC-normalized, lower-cased, diacritics preserved) — the same protocol as the rest of the benchmark.
python
from transformers import WhisperForConditionalGeneration, WhisperProcessorimport torch, soundfile as sf model = WhisperForConditionalGeneration.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-lug")processor = WhisperProcessor.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-lug")wav, sr = sf.read("your_audio.wav") # 16 kHz monofeats = processor(wav, sampling_rate=16000, return_tensors="pt").input_featuresids = model.generate(feats) # multilingual; no forced language tokenprint(processor.batch_decode(ids, skip_special_tokens=True))Part of the WAXAL ASR Benchmark (arXiv:2606.02375).
bibtex
@article{waxalnet2026, title = {The WAXAL ASR Benchmark: Fine-Tuned Edge Models Across 19 African Languages}, author = {Olufemi, Victor Tolulope and Babatunde, Oreoluwa and Njema, Ramsey and others}, year = {2026}, note = {arXiv preprint arXiv:2606.02375}}Supported by Lynguallabs, Open Token, and CMU Africa.
from transformers import WhisperForConditionalGeneration, WhisperProcessorimport torch, soundfile as sf model = WhisperForConditionalGeneration.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-lug")processor = WhisperProcessor.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-lug")wav, sr = sf.read("your_audio.wav") # 16 kHz monofeats = processor(wav, sampling_rate=16000, return_tensors="pt").input_featuresids = model.generate(feats) # multilingual; no forced language tokenprint(processor.batch_decode(ids, skip_special_tokens=True))@article{waxalnet2026, title = {The WAXAL ASR Benchmark: Fine-Tuned Edge Models Across 19 African Languages}, author = {Olufemi, Victor Tolulope and Babatunde, Oreoluwa and Njema, Ramsey and others}, year = {2026}, note = {arXiv preprint arXiv:2606.02375}}