Samyak005

whisper-large-v3-hindi

README

License: apache-2.0

Results (held-out test set, 500 utterances, 60 min of audio)

Table with columns: Metric, Baseline (zero-shot), Fine-tuned
Metric	Baseline (zero-shot)	Fine-tuned
WER	33.17	18.27
CER	19.84	8.99
WER (raw)	37.10	18.23
Substitutions %	23.16	11.82
Deletions %	7.93	4.89
Insertions %	2.08	1.56
BLEU	29.23	41.82
chrF++	52.32	62.88

WER/CER (lower is better) use an Indic-safe normalizer: NFC, punctuation and danda removed, matras kept. Whisper's BasicTextNormalizer strips Devanagari vowel signs and would corrupt Hindi WER. BLEU/chrF++ (higher is better) are computed with sacrebleu on the English translations. Decoding is greedy for both models.

By source dataset

Table with columns: Group, n, WER base → FT, CER base → FT, BLEU base → FT
Group	n	WER base → FT	CER base → FT	BLEU base → FT
IndicVoices-ST	155	33.21 → 21.43	18.57 → 11.27	27.25 → 47.15
Mann-ki-Baat	171	33.26 → 17.35	17.05 → 8.09	27.22 → 32.33
Spoken-Tutorial	174	32.95 → 13.70	25.85 → 6.21	33.70 → 44.88

By clip duration

Table with columns: Group, n, WER base → FT, CER base → FT, BLEU base → FT
Group	n	WER base → FT	CER base → FT	BLEU base → FT
<5 s	203	33.33 → 16.49	21.54 → 7.57	29.85 → 41.20
5-10 s	179	31.89 → 14.87	17.11 → 5.80	31.33 → 42.82
10-20 s	102	32.25 → 16.78	18.75 → 7.39	26.72 → 40.64
20-30 s	16	40.70 → 39.01	30.56 → 29.37	27.71 → 42.23

headline_metrics

error_breakdown

wer_by_source

wer_by_duration

training_curves

Usage

python

import torchfrom transformers import pipeline asr = pipeline("automatic-speech-recognition", model="Samyak005/whisper-large-v3-hindi",               dtype=torch.float16, device="cuda:0") hindi   = asr("audio.wav", generate_kwargs={"language": "hi", "task": "transcribe"})["text"]english = asr("audio.wav", generate_kwargs={"language": "hi", "task": "translate"})["text"]

Always pass language="hi" and the task. For audio longer than 30 s add chunk_length_s=30 to the pipeline.

Training

Table

Data	3,000 Hindi pairs per source with `en_mining_score >= 0.7`, clips <= 30 s. Split 8,200 / 300 / 500 (train / val / test)
Multitask	each training sample is randomly assigned transcribe (Hindi target) or translate (English target)
Hardware	1x A100 80 GB (Colab), bf16, gradient checkpointing
Optimizer	AdamW 8-bit, LR 1e-5, 10% warmup, effective batch 16, 3 epochs (1,539 steps)
Regularization	SpecAugment (`mask_time_prob=0.05`)
Model selection	lowest validation loss (evaluated every 250 steps)
Weights	saved in fp16

Limitations

Clips of 20–30 s barely improve on WER. They are rare in the training data.
Code-mixed technical speech: the references write English words in Latin or Devanagari script inconsistently.
Occasional word-joining errors (e.g. बहुतबहुत).
Trained on 3 domains (conversational, software tutorials, radio addresses) and may not generalize to other ones.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Samyak005

Model Tree

Base

openai/whisper-large-v3

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality