joel-pawar

whisper-marathi-0.2B-Finetuned

Results (FLEURS Marathi test split, 1015 clips)

Table with columns: Metric, Whisper-small (zero-shot), This model
Metric	Whisper-small (zero-shot)	This model
WER % (lower is better)	125.9	43.8
CER % (lower is better)	57.4	13.3
Word-level F1 % (higher is better)	12.9	59.6
Exact sentence match %	0.0	0.4

Notes on how to read these numbers:

Both models were decoded the same way: greedy search, max 300 new tokens, fp16. The baseline was not given any protection against repetition loops, which inflates its WER (it can exceed 100%).
Text was normalised before scoring (Unicode NFC, lowercase, punctuation and symbols removed), so these numbers are not directly comparable to leaderboard figures that use other normalisers.
Word-level F1 and exact match are supplementary metrics. Speech recognition is not a classification task, so WER and CER are the main metrics.

WER and CER Training curve Per-clip WER distribution WER by clip duration Supplementary metrics

How to use

python

from transformers import pipeline asr = pipeline(    "automatic-speech-recognition",    model="joel-pawar/whisper-small-marathi-fleurs",    chunk_length_s=30,    generate_kwargs={"language": "marathi", "task": "transcribe"},)print(asr("your_audio.wav")["text"])

A smoke test is included: python test_model.py (or python test_model.py my_audio.wav).

Training

Base model: openai/whisper-small
Training data: FLEURS Marathi (mr_in) train split, through a parquet copy on Hugging Face (Srijan-Upadhyay/marathi-tts-fleurs). About 12 hours of clean read speech, with clips whose label sequence was too long for Whisper filtered out.
Hyperparameters: learning rate 1e-5, effective batch size 16 (8 x 2 gradient accumulation), 50 warmup steps, 800 steps, fp16, gradient checkpointing
Hardware: Tesla T4
Library versions: transformers 5.19.0, torch 2.11.0+cu130

All raw results are in results/ (per-clip predictions, the 30 worst errors, training log, metrics.json).

Limitations

Trained and tested only on FLEURS read speech: clean audio, one speaker at a time, a limited vocabulary and topics. Accuracy on phone calls, noisy audio, dialects, conversations or overlapping speakers is untested and probably much worse.
A small amount of training data (about 12 hours) and a short training run. Expect errors on rare words, names and numbers.
Output is Devanagari text without punctuation conventions.
Not evaluated for fairness across speakers, gender or dialect.
Not suitable for high-stakes uses such as legal, medical or safety-critical transcription.

License and citations

Released under the MIT license, the same as the base Whisper model. FLEURS is licensed CC BY 4.0.

Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), 2022.
Conneau et al., FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech, 2022.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

joel-pawar

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Results (FLEURS Marathi test split, 1015 clips)

Table with columns: Metric, Whisper-small (zero-shot), This model
Metric	Whisper-small (zero-shot)	This model
WER % (lower is better)	125.9	43.8
CER % (lower is better)	57.4	13.3
Word-level F1 % (higher is better)	12.9	59.6
Exact sentence match %	0.0	0.4

Notes on how to read these numbers:

Both models were decoded the same way: greedy search, max 300 new tokens, fp16. The baseline was not given any protection against repetition loops, which inflates its WER (it can exceed 100%).
Text was normalised before scoring (Unicode NFC, lowercase, punctuation and symbols removed), so these numbers are not directly comparable to leaderboard figures that use other normalisers.
Word-level F1 and exact match are supplementary metrics. Speech recognition is not a classification task, so WER and CER are the main metrics.

WER and CER Training curve Per-clip WER distribution WER by clip duration Supplementary metrics

How to use

python

from transformers import pipeline asr = pipeline(    "automatic-speech-recognition",    model="joel-pawar/whisper-small-marathi-fleurs",    chunk_length_s=30,    generate_kwargs={"language": "marathi", "task": "transcribe"},)print(asr("your_audio.wav")["text"])

from transformers import pipeline asr = pipeline(    "automatic-speech-recognition",    model="joel-pawar/whisper-small-marathi-fleurs",    chunk_length_s=30,    generate_kwargs={"language": "marathi", "task": "transcribe"},)print(asr("your_audio.wav")["text"])

A smoke test is included: python test_model.py (or python test_model.py my_audio.wav).

Training

Base model: openai/whisper-small
Training data: FLEURS Marathi (mr_in) train split, through a parquet copy on Hugging Face (Srijan-Upadhyay/marathi-tts-fleurs). About 12 hours of clean read speech, with clips whose label sequence was too long for Whisper filtered out.
Hyperparameters: learning rate 1e-5, effective batch size 16 (8 x 2 gradient accumulation), 50 warmup steps, 800 steps, fp16, gradient checkpointing
Hardware: Tesla T4
Library versions: transformers 5.19.0, torch 2.11.0+cu130

All raw results are in results/ (per-clip predictions, the 30 worst errors, training log, metrics.json).

Limitations

Trained and tested only on FLEURS read speech: clean audio, one speaker at a time, a limited vocabulary and topics. Accuracy on phone calls, noisy audio, dialects, conversations or overlapping speakers is untested and probably much worse.
A small amount of training data (about 12 hours) and a short training run. Expect errors on rare words, names and numbers.
Output is Devanagari text without punctuation conventions.
Not evaluated for fairness across speakers, gender or dialect.
Not suitable for high-stakes uses such as legal, medical or safety-critical transcription.

License and citations

Released under the MIT license, the same as the base Whisper model. FLEURS is licensed CC BY 4.0.

Radford et al., Robust Speech Recognition via Large-Scale Weak Supervision (Whisper), 2022.
Conneau et al., FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech, 2022.

whisper-marathi-0.2B-Finetuned

README

Results (FLEURS Marathi test split, 1015 clips)

How to use

Training

Limitations

License and citations

Explore FriendliAI today

README

Results (FLEURS Marathi test split, 1015 clips)

How to use

Training

Limitations

License and citations