GLM-5.3 is live. Run Z.ai's latest model on Friendli Model APIs. Try it today ➜
waxal-benchmarking
Available on FriendliAI
Run this model inference on single tenant GPU with unmatched speed and reliability at scale.
Run this model inference with full control and performance in your environment.
Model Details
Model Provider
waxal-benchmarking
Model Tree
openai/whisper-small
Input Modalities
Output Modalities
Supported Functionality
WAXAL's headline finding is that in-language data is essential. Whisper (unlike Omnilingual) does not cover most WAXAL languages in pretraining, so it lets us ask a clean question with no forgetting confound: can training on sibling languages build a language the model never saw? Here Fula is a genealogical isolate in this set — no relatives among the other 18.
| Setting | WER | CER |
|---|---|---|
| Whisper zero-shot (base) | Unsupported (not in Whisper's 99 languages) | — |
| This model (trained on 18 siblings, Fula held out) | 131.7 | 77.6 |
| Monolingual Whisper-Small (trained on Fula) | 42.6 | — |
The gap is stark: sibling training lands far above the monolingual model — training on neighbours cannot substitute for in-language data, even for a genuinely unseen language.
Whisper-Small fine-tuned on the pooled train splits of the 18 non-Fula WAXAL
languages (google/WaxalNLP),
balanced by capping each language, multilingual (no forced language token). 16 kHz mono;
transcripts NFC-normalized + lower-cased, punctuation removed, diacritics preserved.
| Hyperparameter | Value |
|---|---|
| Base | openai/whisper-small (244M) |
| Training languages | 18 (all WAXAL except Fula) |
| Utterances / language | 3,000 (balanced) |
| Steps | 4,000 |
| Optimizer | AdamW, lr 1e-5, 200 warmup |
| Batch size | 16 |
| Precision | fp16 |
| Hardware | 1× H200 |
Languages it was trained on: Acholi, Akan, Amharic, Dagbani, Dagaare, Ewe, Ikposo, Lingala, Luganda, Masaaba, Malagasy, Nyankole, Oromo, Sidama, Shona, Soga, Tigrinya, Wolaytta.
Scored on the held-out Fula test split (jiwer WER/CER, NFC-normalized, lower-cased, diacritics preserved) — the same protocol as the rest of the benchmark.
python
from transformers import WhisperForConditionalGeneration, WhisperProcessorimport torch, soundfile as sf model = WhisperForConditionalGeneration.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-ful")processor = WhisperProcessor.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-ful")wav, sr = sf.read("your_audio.wav") # 16 kHz monofeats = processor(wav, sampling_rate=16000, return_tensors="pt").input_featuresids = model.generate(feats) # multilingual; no forced language tokenprint(processor.batch_decode(ids, skip_special_tokens=True))Part of the WAXAL ASR Benchmark (arXiv:2606.02375).
bibtex
@article{waxalnet2026, title = {The WAXAL ASR Benchmark: Fine-Tuned Edge Models Across 19 African Languages}, author = {Olufemi, Victor Tolulope and Babatunde, Oreoluwa and Njema, Ramsey and others}, year = {2026}, note = {arXiv preprint arXiv:2606.02375}}Supported by Lynguallabs, Open Token, and CMU Africa.
from transformers import WhisperForConditionalGeneration, WhisperProcessorimport torch, soundfile as sf model = WhisperForConditionalGeneration.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-ful")processor = WhisperProcessor.from_pretrained("waxal-benchmarking/whisper-small-waxal-loo-ful")wav, sr = sf.read("your_audio.wav") # 16 kHz monofeats = processor(wav, sampling_rate=16000, return_tensors="pt").input_featuresids = model.generate(feats) # multilingual; no forced language tokenprint(processor.batch_decode(ids, skip_special_tokens=True))@article{waxalnet2026, title = {The WAXAL ASR Benchmark: Fine-Tuned Edge Models Across 19 African Languages}, author = {Olufemi, Victor Tolulope and Babatunde, Oreoluwa and Njema, Ramsey and others}, year = {2026}, note = {arXiv preprint arXiv:2606.02375}}