hlasith

whisper-sinhala-small

README

License: other

Key Improvements

Table with columns: Metric, Base whisper-small, Fine-tuned, Improvement
Metric	Base `whisper-small`	Fine-tuned	Improvement
WER (Word Error Rate)	130.86%	61.73%	−69.13 pp (45% relative)
CER (Character Error Rate)	244.38%	14.86%	−229.52 pp (94% relative)

Evaluated on 10 validation audio samples from the Sinhala speech corpus.

Why Fine-tuning is Essential

The base Whisper-small model was not trained on Sinhala and produces unusable output — mixing Greek, Telugu, Tamil, Arabic, and Hindi characters with English phrases on Sinhala audio. Fine-tuning transforms this into meaningful Sinhala transcription.

Example (Sinhala song vocals, ~4.7 min):

Table with columns: Transcript
	Transcript
Base whisper-small	`ὁ ὁ ὁ ... వార్ల్ల్ల్ ... I'm running, running, running ... My dear, my dear ... Thank you very much for your time ...` (gibberish: Greek, Telugu, English, Arabic, Tamil)
Fine-tuned Sinhala	`සුවදට මද නළයි විසික්කල්‍ය සිවදට මද නළයි විසික්කල්‍ය ... සමුදුරු‍රපටු පොල රැලින්‍යේ ... හාමෝයේ වෝපේ ...` (readable Sinhala with correct phonetic patterns)

Model Details

Table with columns: Property, Value
Property	Value
Base model	`openai/whisper-small`
Architecture	`WhisperForConditionalGeneration`
Parameters	~244M (12 encoder, 12 decoder, d_model=768)
Language	Sinhala (si-LK)
Task	Automatic Speech Recognition (transcription)
Training steps	5,000
Learning rate	1e-4
Batch size

Dataset

Fine-tuned on the Sinhala speech corpus (audio_data/si_lk.lines.txt):

Format: ( audio_id "transcript" ) per line
Audio: 16 kHz mono WAV files
Split: 90% training / 10% validation (random seed 42)
Training samples: ~10,000+
Validation samples: 10 samples used for WER/CER evaluation

Quick Start

Transcribe audio with the pipeline

python
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
import torch

model_id = "harshanaDisa/whisper-sinhala-small"

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float32

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    device=device,
    torch_dtype=torch_dtype,
    chunk_length_s=30,
    stride_length_s=5,
)

result = pipe("path/to/audio.mp3")
print(result["text"])

Command-line usage (local)

bash
# Clone repo and transcribe
git clone https://github.com/harshanaDisa/whisper-sinhala.git
cd whisper-sinhala
python transcribe_vocals.py

Training

bash
# Install dependencies
python -m venv .venv && source .venv/bin/activate
pip install torch transformers accelerate soundfile librosa jiwer datasets pandas

# Prepare dataset
python create_samples.py

# Train
python train.py

# Evaluate (fine-tuned only)
python test_model.py

# Compare against base whisper-small
python test_model.py --compare

Limitations

Domain-specific: Fine-tuned on a specific Sinhala speech corpus; performance may vary on out-of-domain audio (e.g., news, technical content)
WER still ~62%: The error rate is significantly better than the base model but not perfect — further training or a larger dataset would help
No timestamps: The model is configured without timestamp tokens (50363) for cleaner transcription output
Single language: Optimized for Sinhala only; multilingual capabilities of the base Whisper model are degraded

Reproduction

The full training code, dataset preparation scripts, and evaluation pipeline are available in the GitHub repository.

License

This model is released for research and educational purposes. The base Whisper architecture is by OpenAI; the fine-tuned weights are derived from openai/whisper-small.

Citation

bibtex
@inproceedings{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and others},
  booktitle={ICML},
  year={2023}
}

Explore FriendliAI today

Get started Talk to an engineer

README

License: other

Key Improvements

Table with columns: Metric, Base whisper-small, Fine-tuned, Improvement
Metric	Base `whisper-small`	Fine-tuned	Improvement
WER (Word Error Rate)	130.86%	61.73%	−69.13 pp (45% relative)
CER (Character Error Rate)	244.38%	14.86%	−229.52 pp (94% relative)

Evaluated on 10 validation audio samples from the Sinhala speech corpus.

Why Fine-tuning is Essential

Example (Sinhala song vocals, ~4.7 min):

Table with columns: Transcript
	Transcript
Base whisper-small	`ὁ ὁ ὁ ... వార్ల్ల్ల్ ... I'm running, running, running ... My dear, my dear ... Thank you very much for your time ...` (gibberish: Greek, Telugu, English, Arabic, Tamil)
Fine-tuned Sinhala	`සුවදට මද නළයි විසික්කල්‍ය සිවදට මද නළයි විසික්කල්‍ය ... සමුදුරු‍රපටු පොල රැලින්‍යේ ... හාමෝයේ වෝපේ ...` (readable Sinhala with correct phonetic patterns)

Model Details

Table with columns: Property, Value
Property	Value
Base model	`openai/whisper-small`
Architecture	`WhisperForConditionalGeneration`
Parameters	~244M (12 encoder, 12 decoder, d_model=768)
Language	Sinhala (si-LK)
Task	Automatic Speech Recognition (transcription)
Training steps	5,000
Learning rate	1e-4
Batch size

Dataset

Fine-tuned on the Sinhala speech corpus (audio_data/si_lk.lines.txt):

Format: ( audio_id "transcript" ) per line
Audio: 16 kHz mono WAV files
Split: 90% training / 10% validation (random seed 42)
Training samples: ~10,000+
Validation samples: 10 samples used for WER/CER evaluation

Quick Start

Transcribe audio with the pipeline

python
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
import torch

model_id = "harshanaDisa/whisper-sinhala-small"

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float32

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, device_map="auto"
)
processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    device=device,
    torch_dtype=torch_dtype,
    chunk_length_s=30,
    stride_length_s=5,
)

result = pipe("path/to/audio.mp3")
print(result["text"])

Command-line usage (local)

bash
# Clone repo and transcribe
git clone https://github.com/harshanaDisa/whisper-sinhala.git
cd whisper-sinhala
python transcribe_vocals.py

Training

bash
# Install dependencies
python -m venv .venv && source .venv/bin/activate
pip install torch transformers accelerate soundfile librosa jiwer datasets pandas

# Prepare dataset
python create_samples.py

# Train
python train.py

# Evaluate (fine-tuned only)
python test_model.py

# Compare against base whisper-small
python test_model.py --compare

Limitations

Domain-specific: Fine-tuned on a specific Sinhala speech corpus; performance may vary on out-of-domain audio (e.g., news, technical content)
WER still ~62%: The error rate is significantly better than the base model but not perfect — further training or a larger dataset would help
No timestamps: The model is configured without timestamp tokens (50363) for cleaner transcription output
Single language: Optimized for Sinhala only; multilingual capabilities of the base Whisper model are degraded

Reproduction

The full training code, dataset preparation scripts, and evaluation pipeline are available in the GitHub repository.

License

This model is released for research and educational purposes. The base Whisper architecture is by OpenAI; the fine-tuned weights are derived from openai/whisper-small.

Citation

bibtex
@inproceedings{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford, Alec and others},
  booktitle={ICML},
  year={2023}
}

whisper-sinhala-small

README

Key Improvements

Why Fine-tuning is Essential

Model Details

Dataset

Quick Start

Transcribe audio with the pipeline

Command-line usage (local)

Training

Limitations

Reproduction

License

Citation

Links

Explore FriendliAI today

README

Key Improvements

Why Fine-tuning is Essential

Model Details

Dataset

Quick Start

Transcribe audio with the pipeline

Command-line usage (local)

Training

Limitations

Reproduction

License

Citation

Links