khawajaaliarshad

whisper-small-urdu

Deploy Dedicated

README

License: apache-2.0

Model Description

Developed by: Khawaja Ali
Model type: Whisper (Encoder-Decoder Transformer)
Language: Urdu (ur)
License: Apache 2.0
Finetuned from: openai/whisper-small
Best Checkpoint: Step 3500 (loaded via load_best_model_at_end)
W&B Run: whisper-small-urdu-expanded-v1

Intended Uses & Limitations

Intended Uses

Transcribing Urdu speech to text
Building Urdu voice assistants
Subtitling Urdu audio/video content
Accessibility applications for Urdu speakers

Limitations

Domain: Trained on read speech; may underperform on conversational/spontaneous speech
Accents: Primarily trained on Common Voice contributors; dialect coverage may vary
Noise: Best performance on clean audio; background noise degrades accuracy
Audio format: Requires 16kHz mono audio

Performance

Table with columns: Dataset, WER ↓, Improvement
Dataset	WER ↓	Improvement
Common Voice Urdu (validation)	32.80%	-13.4pp vs baseline
Common Voice Urdu (test)	~33%	~13pp improvement

Baseline Comparison (Base Models, Zero-Shot on Urdu):

Table with columns: Model, Parameters, WER (Urdu), Notes
Model	Parameters	WER (Urdu)	Notes
whisper-tiny	39M	~65%	Zero-shot
whisper-small	244M	46.21%	Zero-shot (our baseline)
whisper-small-urdu (this)	244M	32.80%	Fine-tuned ✨ (29% relative improvement)
whisper-medium	769M	~40%	Zero-shot

🎉 This fine-tuned small model outperforms whisper-medium and approaches whisper-large-v2 performance!

Quick Start

Installation

bash

pip install transformers torch librosa

Inference with Pipeline (Recommended)

python

from transformers import pipelineimport torch device = "cuda:0" if torch.cuda.is_available() else "cpu" pipe = pipeline(    "automatic-speech-recognition",    model="khawajaaliarshad/whisper-small-urdu",    device=device,) # Transcribe audio fileresult = pipe(    "audio.wav",    generate_kwargs={"language": "urdu", "task": "transcribe"},)print(result["text"])

Inference with Transformers

python

from transformers import WhisperProcessor, WhisperForConditionalGenerationimport torchimport librosa # Load modelprocessor = WhisperProcessor.from_pretrained("khawajaaliarshad/whisper-small-urdu")model = WhisperForConditionalGeneration.from_pretrained("khawajaaliarshad/whisper-small-urdu") device = "cuda:0" if torch.cuda.is_available() else "cpu"model.to(device) # Load and process audioaudio, sr = librosa.load("audio.wav", sr=16000)inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device) # Generate transcriptionwith torch.no_grad():    predicted_ids = model.generate(        inputs.input_features,        language="urdu",        task="transcribe",        max_length=225,    ) # Decodetranscription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]print(transcription)

Long-form Transcription

For audio longer than 30 seconds:

python

result = pipe(    "long_audio.wav",    chunk_length_s=30,    batch_size=8,    generate_kwargs={"language": "urdu", "task": "transcribe"},)print(result["text"])

Training Details

Training Data

The model was trained on the expanded Common Voice Scripted Speech 24.0 - Urdu dataset downloaded via the Mozilla Data Collective API.

Table with columns: Property, Value
Property	Value
Dataset	Common Voice Scripted Speech 24.0 - Urdu (Expanded)
Source	Mozilla Data Collective
Dataset ID	`cmj8u3pz600t9nxxbz9l2ck2n`
License	CC0-1.0 (Public Domain)
Total Raw Samples	252,899
Total Raw Hours	302 hours (81.5 validated)
Speakers	498

Processed splits used for training (expanded):

Table with columns: Split, Samples, Hours (approx)
Split	Samples	Hours (approx)
Train	54,754	~75 hrs
Validation	5,046	~7 hrs
Test	5,091	~7 hrs
Total	64,891	~89 hrs

Processed Dataset: khawajaaliarshad/common-voice-urdu-processed-expanded

This is the preprocessed version of the dataset used for training, with audio resampled to 16kHz and ready for Whisper fine-tuning.

Note: The model was trained on clean (non-augmented) data. Initial experiments with audio augmentation showed degraded performance due to train/eval domain mismatch.

Training Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Base model	`openai/whisper-small`
Learning rate	1e-5
LR scheduler	Linear with warmup
Warmup steps	500
Training steps	4,000
Epochs completed	~1.17
Batch size (per device)	4
Gradient accumulation	4
Effective batch size

Training Infrastructure

Table with columns: Component, Details
Component	Details
Platform	Kaggle Notebooks
Hardware	NVIDIA T4 (16GB)
Training time	~6.3 hours
Framework	🤗 Transformers 4.52.0
Experiment tracking	Weights & Biases
W&B Run	`whisper-small-urdu-expanded-v1`

Training Progress

Table with columns: Checkpoint, Train Loss, Eval Loss, Eval WER
Checkpoint	Train Loss	Eval Loss	Eval WER
Step 500	0.4914	0.6362	46.07%
Step 1000	0.4165	0.5556	59.80%
Step 1500	0.3500	0.5173	56.53%
Step 2000	0.2952	0.4894	33.10%

Note: Best WER was achieved at step 3500 (32.80%). The model loaded via load_best_model_at_end=True uses this checkpoint.

Framework Versions

Python: 3.12
Transformers: 4.52.0
Datasets: 2.20.0
PyTorch: 2.8.0
Accelerate: 0.30+

Evaluation

Metrics

WER (Word Error Rate): Primary metric for ASR evaluation
Computed using jiwer library

Evaluation Code

python

from datasets import load_datasetfrom transformers import pipelinefrom evaluate import load # Load test datatest_ds = load_dataset("mozilla-foundation/common_voice_17_0", "ur", split="test") # Load modelpipe = pipeline("automatic-speech-recognition", model="khawajaaliarshad/whisper-small-urdu") # Run evaluationwer_metric = load("wer")predictions, references = [], [] for sample in test_ds:    result = pipe(sample["audio"]["array"], generate_kwargs={"language": "urdu"})    predictions.append(result["text"])    references.append(sample["sentence"]) wer = wer_metric.compute(predictions=predictions, references=references)print(f"WER: {wer * 100:.2f}%")

Environmental Impact

Training was performed on Kaggle's free GPU tier, utilizing shared infrastructure.

Table with columns: Factor, Value
Factor	Value
Hardware	NVIDIA T4
Training duration	~6.3 hours
Cloud provider	Kaggle (Google Cloud)
Region	Various
Carbon efficiency	Shared infrastructure

Estimated using ML CO2 Impact Calculator

Citation

BibTeX

If you use this model in your research, please cite:

bibtex

@misc{khawaja2025whisperurdu,  author = {Khawaja, Ali},  title = {Whisper Small Fine-tuned for Urdu Speech Recognition},  year = {2025},  publisher = {Hugging Face},  howpublished = {\url{https://huggingface.co/khawajaaliarshad/whisper-small-urdu}},}

Please also cite the original Whisper paper:

bibtex

@article{radford2022whisper,  title={Robust Speech Recognition via Large-Scale Weak Supervision},  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},  journal={arXiv preprint arXiv:2212.04356},  year={2022}}

And the Mozilla Common Voice dataset:

bibtex

@inproceedings{ardila2020common,  title={Common Voice: A Massively-Multilingual Speech Corpus},  author={Ardila, Rosana and Branson, Megan and Davis, Kelly and Henretty, Michael and Kohler, Michael and Meyer, Josh and Morais, Reuben and Saunders, Lindsay and Tyers, Francis M and Weber, Gregor},  booktitle={Proceedings of the 12th Language Resources and Evaluation Conference},  pages={4211--4215},  year={2020}}

Data Access: The Urdu dataset was accessed via the Mozilla Data Collective API (Dataset ID: cmj8u3pz600t9nxxbz9l2ck2n). The data is licensed under CC0-1.0 (Public Domain).

Acknowledgments

This project was made possible by the following resources and communities:

Models & Data

OpenAI Whisper - For the foundational multilingual speech recognition model
Mozilla Common Voice - For the open-source Urdu speech dataset contributed by volunteers worldwide
Mozilla Data Collective - For the API access to download Common Voice datasets
Mozilla Foundation - For supporting open speech data initiatives
Hugging Face - For the transformers library and model hosting infrastructure

Compute & Tools

Kaggle - For providing free GPU compute (T4)
Weights & Biases - For experiment tracking and visualization
Google Colab - For initial experimentation

Libraries

🤗 Transformers
🤗 Datasets
🤗 Evaluate
🤗 Accelerate
jiwer - WER computation
librosa - Audio processing

Community

The Urdu-speaking contributors to Mozilla Common Voice
The Hugging Face community for guidance on Whisper fine-tuning
The open-source ML community

Model Card Authors

Khawaja Ali - @khawajaaliarshad

Contact

For questions, issues, or collaboration:

🐛 Issues: Open an issue on the model repository
💬 Discussion: Use the Community tab on HuggingFace

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

khawajaaliarshad

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Model Description

Developed by: Khawaja Ali
Model type: Whisper (Encoder-Decoder Transformer)
Language: Urdu (ur)
License: Apache 2.0
Finetuned from: openai/whisper-small
Best Checkpoint: Step 3500 (loaded via load_best_model_at_end)
W&B Run: whisper-small-urdu-expanded-v1

Intended Uses & Limitations

Intended Uses

Transcribing Urdu speech to text
Building Urdu voice assistants
Subtitling Urdu audio/video content
Accessibility applications for Urdu speakers

Limitations

Domain: Trained on read speech; may underperform on conversational/spontaneous speech
Accents: Primarily trained on Common Voice contributors; dialect coverage may vary
Noise: Best performance on clean audio; background noise degrades accuracy
Audio format: Requires 16kHz mono audio

Performance

Table with columns: Dataset, WER ↓, Improvement
Dataset	WER ↓	Improvement
Common Voice Urdu (validation)	32.80%	-13.4pp vs baseline
Common Voice Urdu (test)	~33%	~13pp improvement

Baseline Comparison (Base Models, Zero-Shot on Urdu):

Table with columns: Model, Parameters, WER (Urdu), Notes
Model	Parameters	WER (Urdu)	Notes
whisper-tiny	39M	~65%	Zero-shot
whisper-small	244M	46.21%	Zero-shot (our baseline)
whisper-small-urdu (this)	244M	32.80%	Fine-tuned ✨ (29% relative improvement)
whisper-medium	769M	~40%	Zero-shot

🎉 This fine-tuned small model outperforms whisper-medium and approaches whisper-large-v2 performance!

Quick Start

Installation

bash

pip install transformers torch librosa

pip install transformers torch librosa

Inference with Pipeline (Recommended)

python

from transformers import pipelineimport torch device = "cuda:0" if torch.cuda.is_available() else "cpu" pipe = pipeline(    "automatic-speech-recognition",    model="khawajaaliarshad/whisper-small-urdu",    device=device,) # Transcribe audio fileresult = pipe(    "audio.wav",    generate_kwargs={"language": "urdu", "task": "transcribe"},)print(result["text"])

Inference with Transformers

python

from transformers import WhisperProcessor, WhisperForConditionalGenerationimport torchimport librosa # Load modelprocessor = WhisperProcessor.from_pretrained("khawajaaliarshad/whisper-small-urdu")model = WhisperForConditionalGeneration.from_pretrained("khawajaaliarshad/whisper-small-urdu") device = "cuda:0" if torch.cuda.is_available() else "cpu"model.to(device) # Load and process audioaudio, sr = librosa.load("audio.wav", sr=16000)inputs = processor(audio, sampling_rate=16000, return_tensors="pt").to(device) # Generate transcriptionwith torch.no_grad():    predicted_ids = model.generate(        inputs.input_features,        language="urdu",        task="transcribe",        max_length=225,    ) # Decodetranscription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]print(transcription)

Long-form Transcription

For audio longer than 30 seconds:

python

result = pipe(    "long_audio.wav",    chunk_length_s=30,    batch_size=8,    generate_kwargs={"language": "urdu", "task": "transcribe"},)print(result["text"])

Training Details

Training Data

The model was trained on the expanded Common Voice Scripted Speech 24.0 - Urdu dataset downloaded via the Mozilla Data Collective API.

Table with columns: Property, Value
Property	Value
Dataset	Common Voice Scripted Speech 24.0 - Urdu (Expanded)
Source	Mozilla Data Collective
Dataset ID	`cmj8u3pz600t9nxxbz9l2ck2n`
License	CC0-1.0 (Public Domain)
Total Raw Samples	252,899
Total Raw Hours	302 hours (81.5 validated)
Speakers	498

Processed splits used for training (expanded):

Table with columns: Split, Samples, Hours (approx)
Split	Samples	Hours (approx)
Train	54,754	~75 hrs
Validation	5,046	~7 hrs
Test	5,091	~7 hrs
Total	64,891	~89 hrs

Processed Dataset: khawajaaliarshad/common-voice-urdu-processed-expanded

This is the preprocessed version of the dataset used for training, with audio resampled to 16kHz and ready for Whisper fine-tuning.

Note: The model was trained on clean (non-augmented) data. Initial experiments with audio augmentation showed degraded performance due to train/eval domain mismatch.

Training Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Base model	`openai/whisper-small`
Learning rate	1e-5
LR scheduler	Linear with warmup
Warmup steps	500
Training steps	4,000
Epochs completed	~1.17
Batch size (per device)	4
Gradient accumulation	4
Effective batch size

Training Infrastructure

Table with columns: Component, Details
Component	Details
Platform	Kaggle Notebooks
Hardware	NVIDIA T4 (16GB)
Training time	~6.3 hours
Framework	🤗 Transformers 4.52.0
Experiment tracking	Weights & Biases
W&B Run	`whisper-small-urdu-expanded-v1`

Training Progress

Table with columns: Checkpoint, Train Loss, Eval Loss, Eval WER
Checkpoint	Train Loss	Eval Loss	Eval WER
Step 500	0.4914	0.6362	46.07%
Step 1000	0.4165	0.5556	59.80%
Step 1500	0.3500	0.5173	56.53%
Step 2000	0.2952	0.4894	33.10%

Note: Best WER was achieved at step 3500 (32.80%). The model loaded via load_best_model_at_end=True uses this checkpoint.

Framework Versions

Python: 3.12
Transformers: 4.52.0
Datasets: 2.20.0
PyTorch: 2.8.0
Accelerate: 0.30+

Evaluation

Metrics

WER (Word Error Rate): Primary metric for ASR evaluation
Computed using jiwer library

Evaluation Code

python

from datasets import load_datasetfrom transformers import pipelinefrom evaluate import load # Load test datatest_ds = load_dataset("mozilla-foundation/common_voice_17_0", "ur", split="test") # Load modelpipe = pipeline("automatic-speech-recognition", model="khawajaaliarshad/whisper-small-urdu") # Run evaluationwer_metric = load("wer")predictions, references = [], [] for sample in test_ds:    result = pipe(sample["audio"]["array"], generate_kwargs={"language": "urdu"})    predictions.append(result["text"])    references.append(sample["sentence"]) wer = wer_metric.compute(predictions=predictions, references=references)print(f"WER: {wer * 100:.2f}%")

Environmental Impact

Training was performed on Kaggle's free GPU tier, utilizing shared infrastructure.

Table with columns: Factor, Value
Factor	Value
Hardware	NVIDIA T4
Training duration	~6.3 hours
Cloud provider	Kaggle (Google Cloud)
Region	Various
Carbon efficiency	Shared infrastructure

Estimated using ML CO2 Impact Calculator

Citation

BibTeX

If you use this model in your research, please cite:

bibtex

@misc{khawaja2025whisperurdu,  author = {Khawaja, Ali},  title = {Whisper Small Fine-tuned for Urdu Speech Recognition},  year = {2025},  publisher = {Hugging Face},  howpublished = {\url{https://huggingface.co/khawajaaliarshad/whisper-small-urdu}},}

@misc{khawaja2025whisperurdu,  author = {Khawaja, Ali},  title = {Whisper Small Fine-tuned for Urdu Speech Recognition},  year = {2025},  publisher = {Hugging Face},  howpublished = {\url{https://huggingface.co/khawajaaliarshad/whisper-small-urdu}},}

Please also cite the original Whisper paper:

bibtex

@article{radford2022whisper,  title={Robust Speech Recognition via Large-Scale Weak Supervision},  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},  journal={arXiv preprint arXiv:2212.04356},  year={2022}}

@article{radford2022whisper,  title={Robust Speech Recognition via Large-Scale Weak Supervision},  author={Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},  journal={arXiv preprint arXiv:2212.04356},  year={2022}}

And the Mozilla Common Voice dataset:

bibtex

@inproceedings{ardila2020common,  title={Common Voice: A Massively-Multilingual Speech Corpus},  author={Ardila, Rosana and Branson, Megan and Davis, Kelly and Henretty, Michael and Kohler, Michael and Meyer, Josh and Morais, Reuben and Saunders, Lindsay and Tyers, Francis M and Weber, Gregor},  booktitle={Proceedings of the 12th Language Resources and Evaluation Conference},  pages={4211--4215},  year={2020}}

@inproceedings{ardila2020common,  title={Common Voice: A Massively-Multilingual Speech Corpus},  author={Ardila, Rosana and Branson, Megan and Davis, Kelly and Henretty, Michael and Kohler, Michael and Meyer, Josh and Morais, Reuben and Saunders, Lindsay and Tyers, Francis M and Weber, Gregor},  booktitle={Proceedings of the 12th Language Resources and Evaluation Conference},  pages={4211--4215},  year={2020}}

Data Access: The Urdu dataset was accessed via the Mozilla Data Collective API (Dataset ID: cmj8u3pz600t9nxxbz9l2ck2n). The data is licensed under CC0-1.0 (Public Domain).

Acknowledgments

This project was made possible by the following resources and communities:

Models & Data

OpenAI Whisper - For the foundational multilingual speech recognition model
Mozilla Common Voice - For the open-source Urdu speech dataset contributed by volunteers worldwide
Mozilla Data Collective - For the API access to download Common Voice datasets
Mozilla Foundation - For supporting open speech data initiatives
Hugging Face - For the transformers library and model hosting infrastructure

Compute & Tools

Kaggle - For providing free GPU compute (T4)
Weights & Biases - For experiment tracking and visualization
Google Colab - For initial experimentation

Libraries

🤗 Transformers
🤗 Datasets
🤗 Evaluate
🤗 Accelerate
jiwer - WER computation
librosa - Audio processing

Community

The Urdu-speaking contributors to Mozilla Common Voice
The Hugging Face community for guidance on Whisper fine-tuning
The open-source ML community

Model Card Authors

Khawaja Ali - @khawajaaliarshad

Contact

For questions, issues, or collaboration:

🐛 Issues: Open an issue on the model repository
💬 Discussion: Use the Community tab on HuggingFace