Results
Table with columns: Model, Test set, Clips, WER raw, WER normalized| Model | Test set | Clips | WER raw | WER normalized |
|---|
| openai/whisper-small | Common Voice 27.0 Pashto | 1,500 | 106.0 | 104.2 |
| whisper-small-pashto | Common Voice 27.0 Pashto | 5,000 | 33.1 | 27.8 |
| openai/whisper-small | Common Voice Spontaneous Speech 5.0 Pashto | 76 | 93.4 | 91.4 |
| whisper-small-pashto | Common Voice Spontaneous Speech 5.0 Pashto | 76 | 40.3 | 33.0 |
Normalized WER ignores punctuation, diacritics and letters Pashto writers use interchangeably (ي/ی, ك/ک, ګ/گ, Arabic-Indic digits), so two spellings of the same words do not count as an error. The untuned model often writes Pashto in Latin letters or adds words, which puts its WER above 100. The spontaneous-speech set is unscripted speech the model was not trained on.
During training, on 500 dev clips: loss 0.3107, raw WER 29.10.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 500
- training_steps: 5000
- mixed_precision_training: Native AMP
Training results
Table with columns: Training Loss, Epoch, Step, Validation Loss, Wer| Training Loss | Epoch | Step | Validation Loss | Wer |
|---|
| 0.8218 | 0.1350 | 1000 | 0.4323 | 39.4820 |
| 0.6801 | 0.2699 | 2000 | 0.3679 | 32.7877 |
| 0.5982 | 0.4049 | 3000 | 0.3349 | 29.2206 |
| 0.5473 | 0.5398 | 4000 | 0.3197 | 29.9291 |
| 0.5596 | 0.6748 | 5000 | 0.3107 | 29.0985 |
Framework versions
- Transformers 5.21.0
- Pytorch 2.11.0+cu128
- Datasets 4.8.5
- Tokenizers 0.23.3