Dataset

The model has been fine-tuned using cdli/ugandan_luganda_nonstandard_speech_v1.0, a dataset of speech samples of people living with impaired speech across a range of impairment severity levels and etiologies.

Training

The train split was used for training, and the dev split for selecting the best checkpoint.

This Whisper model was fine-tuned and is decoded using the Swahili (sw) language setting — out of all languages Whisper supports, the one most similar to Luganda.

All model parameters (encoder, decoder, and output projection) were fine-tuned, with SpecAugment enabled.

Evaluation

This model was evaluated on the test split of the dataset. Utterances longer than 30 seconds were excluded:

• Examples evaluated: 1028
• Speakers: 9

For decoding we ran Whisper with language=sw, task=transcribe, greedy search (num_beams=1, do_sample=False).

Results are compared against the unadapted base model cdli/whisper-large-v3_finetuned_ugandan_luganda_waxal_7_standard_speech_v1.0, evaluated identically, to show the effect of fine-tuning on non-standard speech.

We report two complementary word error rate (WER) metrics, both computed on text normalized with Whisper's BasicTextNormalizer:

• Standard (corpus-level) WER — the usual error rate, pooling all reference words and edit errors across the entire test set.
• Per-utterance averaged WER — WER computed separately for each utterance, each capped at 1.0, then averaged across utterances.

The per-utterance averaged WER bounds each utterance to [0, 1] and weights all utterances equally, so it reflects typical performance without a few catastrophic utterances dominating — but it is not a true error rate and isn't directly comparable to other published WER, hence we report the standard, corpus-level WER as well.

Results

Overall Results

• Adapted — this CDLI model, fine-tuned on non-standard speech.
• Unadapted — the base model it was fine-tuned from (here: cdli/whisper-large-v3_finetuned_ugandan_luganda_waxal_7_standard_speech_v1.0).

Detailed Analysis

Aggregated results can hide important underlying patterns, so we also break the WER down by subset: per speaker, and — where speaker severity is available — per impairment severity group.

Results by impairment severity

All WER values below are the per-utterance averaged WER, first averaged per speaker and then averaged within each severity group. n_speakers and n_utterances are the number of speakers and test utterances in each group.

Results by speaker

Per-utterance averaged WER per speaker. n_utterances is the number of test utterances for that speaker.