Model Details
- Model type: Whisper encoder-decoder Transformer
- Base model:
openai/whisper-small
- Task: Automatic Speech Recognition
- Language: Arabic
- Target speech variety: Yemeni Arabic
- Framework: Hugging Face Transformers
- Model author:
tahaalselwii
Results
The pretrained base model and the fine-tuned model were evaluated on
the same held-out custom test split.
Table with columns: Model, Test loss, WER (%)| Model | Test loss | WER (%) |
|---|
openai/whisper-small | 3.6458 | 84.3240 |
tahaalselwii/whisper-small-yemeni | 0.9923 | 63.4929 |
The fine-tuned model achieved a 24.70% relative improvement, with WER decreasing from 84.32% to 63.49%, a reduction of 20.83 percentage points.
Usage
pip install -U torch transformers accelerate "datasets[audio]"
import torch
from datasets import Audio, load_dataset
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
model_id = "tahaalselwii/whisper-small-yemeni"
device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id).to(device)
model.eval()
dataset = load_dataset(
"UBC-NLP/Casablanca",
"Yemen",
split="test",
)
dataset = dataset.cast_column(
"audio",
Audio(sampling_rate=16_000),
)
example = dataset[0]
audio = example["audio"]
inputs = processor(
audio["array"],
sampling_rate=audio["sampling_rate"],
return_tensors="pt",
return_attention_mask=True,
)
with torch.no_grad():
predicted_ids = model.generate(
input_features=inputs.input_features.to(device),
attention_mask=inputs.attention_mask.to(device),
language="ar",
task="transcribe",
)
prediction = processor.batch_decode(
predicted_ids,
skip_special_tokens=True,
)[0]
print("Reference:", example["transcription"])
print("Prediction:", prediction)
Training and Evaluation Data
The model was trained using the Yemen configuration of
UBC-NLP/Casablanca. The dataset snapshot used by the training initially contained
1,606 examples across its available source splits. All available source splits were concatenated, shuffled with seed
42, and divided into a new custom split:
Table with columns: Custom split, Percentage, Examples used| Custom split | Percentage | Examples used |
|---|
| Train | 80% | 1,284 |
| Validation | 10% | 161 |
| Test | 10% | 160 |
Limitations
Yemeni Arabic speech data is scarce, and no sufficiently large, suitable dataset was available for this work. As a result, the model was trained and evaluated on a limited dataset, including only 160 test examples. The reported results should therefore be considered preliminary, and performance may vary with new speakers and recording conditions.
Citation
If you use this model, please cite the model repository, the Casablanca dataset paper, and the Whisper paper.
Model
@misc{tahaalselwii2026whisper_yemeni,
author = {Taha Alselwi},
title = {Whisper Small Yemeni},
year = {2026}
}
Training Dataset
@article{talafha2024casablanca,
title = {Casablanca: Data and Models for Multidialectal Arabic Speech Recognition},
author = {Talafha, Bashar and others},
journal = {arXiv preprint arXiv:2410.04527},
year = {2024}
}
Base Model
@article{radford2022robust,
title = {Robust Speech Recognition via Large-Scale Weak Supervision},
author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},
journal = {arXiv preprint arXiv:2212.04356},
year = {2022}
}
Acknowledgements
This model builds on OpenAI Whisper and the Casablanca dataset created
by the UBC Deep Learning and NLP Lab and its contributors.