Evaluation

This model was evaluated on the test split of cdli/kenyan_swahili_nonstandard_speech_v1.0, a non-standard speech dataset for Kenyan Swahili. Examples longer than 30 seconds were excluded. For decoding we ran Whisper with language=sw, task=transcribe, greedy search (num_beams=1, do_sample=False). Results are compared against the unadapted base model cdli/whisper-small-Swahili_finetuned_small_CV20, evaluated identically, to show the effect of fine-tuning on non-standard speech.

• Examples evaluated: 849
• Speakers: 9

Metrics

We report two complementary word error rate (WER) metrics, both computed on text normalized with Whisper's BasicTextNormalizer:

• Standard (corpus-level) WER — the usual error rate, pooling all reference words and edit errors across the entire test set.
• Per-utterance averaged WER — WER computed separately for each utterance, each capped at 1.0, then averaged across utterances.

The per-utterance averaged WER bounds each utterance to [0, 1] and weights all utterances equally, so it reflects typical performance without a few catastrophic utterances dominating — but it is not a true error rate and isn't directly comparable to other published WER, hence we report the standard, corpus-level WER as well.

Overall Results

• Adapted — this CDLI model, fine-tuned on non-standard speech.
• Unadapted — the base model it was fine-tuned from (here: cdli/whisper-small-Swahili_finetuned_small_CV20).

Detailed Analysis

For non-standard speech, aggregated results can hide important underlying patterns. Hence, we also report the WERs for different subsets: per severity group as well as per speaker.

Results by impairment severity

All WER values below are the per-utterance averaged WER, first averaged per speaker and then averaged within each severity group. n_speakers and n_utterances are the number of speakers and test utterances in each group.

Results by speaker

Per-utterance averaged WER per speaker. n_utterances is the number of test utterances for that speaker.