thunderboltc

whisper-small-santali-sanlish

Model Details

Model Description

Developed by: thunderboltc
Model type: Whisper Small (Encoder-Decoder Transformer, Seq2Seq)
Language: Santali (sat) — transcribed in custom Roman transliteration (Sanlish)
Base model: openai/whisper-small
Parameters: 241,734,912 (~242M)
Fine-tuned for: Low-resource ASR — Santali speech to Sanlish text
License: MIT

What is Sanlish?

Sanlish is a custom Roman-script transliteration of Santali designed to bridge the gap between Santali phonetics and Latin-alphabet-based ASR systems. It was developed as part of this thesis to serve as an intermediate representation between Santali audio and Bangla translation.

Training Details

Training Data

Dataset: 502 Santali audio recordings with parallel Sanlish transliteration annotations
Collected by: thunderboltc (thesis dataset, not yet publicly released)
Split:

Table with columns: Split, Size, Percentage
Split	Size	Percentage
Train	401	80%
Validation	50	10%
Test	51	10%

Audio format: WAV, 16kHz mono
Text target: Custom Roman transliteration (Sanlish)

Training Procedure

Preprocessing

Audio resampled to 16,000 Hz
Log-mel spectrogram features extracted using WhisperFeatureExtractor (80 mel bins, 3000 frames)
Text tokenized using WhisperTokenizer with language=english, task=transcribe
forced_decoder_ids and suppress_tokens disabled to allow custom vocabulary transcription

Training Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Epochs	25
Learning Rate	1e-5
Batch Size (per device)	8
Gradient Accumulation Steps	2 (effective batch = 16)
Warmup Steps	50
Precision	fp16 mixed precision
Evaluation Strategy	Per epoch
Best Model Selection	Lowest validation WER
Hardware	NVIDIA Tesla T4 (Google Colab)

Best Checkpoint

Epoch 13 was selected as the best checkpoint based on lowest validation WER (53.41%).

Evaluation

Results on Test Set (51 utterances)

Table with columns: Metric, Value
Metric	Value
Word Error Rate (WER)	56.55%
Character Error Rate (CER)	13.31%

These results are consistent with expectations for a low-resource ASR system trained on ~400 utterances. The large gap between WER and CER reflects that the model captures phonetic character structure well, even when full word sequences deviate from the reference.

Sample Predictions

Table with columns: Reference, Prediction
Reference	Prediction
onde alom chala a	onde alom chala a ✓
tehindo adi raashkaien yaikaoda	tehindo adi reskaien yaikaoda
gapa shetah ale saote baskiyago meh	gapa setah ale saote baskiyago meh
shetay bhuh da	shetay bhuheda

How to Get Started

python
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

# Load model and processor
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-sanlish")
model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-sanlish")

# Load audio (must be 16kHz)
audio, sr = librosa.load("your_santali_audio.wav", sr=16000)

# Preprocess
inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt")

# Generate transcription
with torch.no_grad():
    predicted_ids = model.generate(inputs.input_features)

# Decode
transcription = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
print("Sanlish transcription:", transcription[0])

Intended Use

Direct Use

Transcribing Santali speech into Sanlish (custom Roman transliteration) for downstream NLP tasks.

Downstream Use

This model is designed to feed into a Machine Translation (MT) model (NLLB-200) for end-to-end Santali Speech → Bangla Text translation.

Out-of-Scope Use

Not suitable for general-purpose ASR in other languages
Not recommended for standard Santali Ol Chiki script transcription
Performance may degrade on speaker accents or recording conditions not represented in the 502-utterance training set

Limitations & Bias

Trained on only 502 utterances — a very small dataset by ASR standards
May not generalize well to unseen speakers or recording environments
Sanlish transliteration scheme has inherent inconsistencies (word boundary variation, hyphenation) which inflate WER beyond true phonetic error rate
No speaker diversity information available for the training set

Citation

If you use this model in your research, please cite:

bibtex
@misc{thunderboltc2025whisper-santali,
  author = {thunderboltc},
  title = {Whisper Small Fine-tuned on Santali Custom Transliteration (Sanlish)},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/thunderboltc/whisper-small-santali-sanlish}
}

Environmental Impact

Hardware: NVIDIA Tesla T4
Cloud Provider: Google Colab
Hours used: ~1.25 hours
Carbon emissions: Estimated using the ML Impact Calculator

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

thunderboltc

Model Tree

Base

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer

Model Details

Model Description

Developed by: thunderboltc
Model type: Whisper Small (Encoder-Decoder Transformer, Seq2Seq)
Language: Santali (sat) — transcribed in custom Roman transliteration (Sanlish)
Base model: openai/whisper-small
Parameters: 241,734,912 (~242M)
Fine-tuned for: Low-resource ASR — Santali speech to Sanlish text
License: MIT

What is Sanlish?

Training Details

Training Data

Dataset: 502 Santali audio recordings with parallel Sanlish transliteration annotations
Collected by: thunderboltc (thesis dataset, not yet publicly released)
Split:

Table with columns: Split, Size, Percentage
Split	Size	Percentage
Train	401	80%
Validation	50	10%
Test	51	10%

Audio format: WAV, 16kHz mono
Text target: Custom Roman transliteration (Sanlish)

Training Procedure

Preprocessing

Audio resampled to 16,000 Hz
Log-mel spectrogram features extracted using WhisperFeatureExtractor (80 mel bins, 3000 frames)
Text tokenized using WhisperTokenizer with language=english, task=transcribe
forced_decoder_ids and suppress_tokens disabled to allow custom vocabulary transcription

Training Hyperparameters

Table with columns: Parameter, Value
Parameter	Value
Epochs	25
Learning Rate	1e-5
Batch Size (per device)	8
Gradient Accumulation Steps	2 (effective batch = 16)
Warmup Steps	50
Precision	fp16 mixed precision
Evaluation Strategy	Per epoch
Best Model Selection	Lowest validation WER
Hardware	NVIDIA Tesla T4 (Google Colab)

Best Checkpoint

Epoch 13 was selected as the best checkpoint based on lowest validation WER (53.41%).

Evaluation

Results on Test Set (51 utterances)

Table with columns: Metric, Value
Metric	Value
Word Error Rate (WER)	56.55%
Character Error Rate (CER)	13.31%

Sample Predictions

Table with columns: Reference, Prediction
Reference	Prediction
onde alom chala a	onde alom chala a ✓
tehindo adi raashkaien yaikaoda	tehindo adi reskaien yaikaoda
gapa shetah ale saote baskiyago meh	gapa setah ale saote baskiyago meh
shetay bhuh da	shetay bhuheda

How to Get Started

python
import torch
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa

# Load model and processor
processor = WhisperProcessor.from_pretrained("thunderboltc/whisper-small-santali-sanlish")
model = WhisperForConditionalGeneration.from_pretrained("thunderboltc/whisper-small-santali-sanlish")

# Load audio (must be 16kHz)
audio, sr = librosa.load("your_santali_audio.wav", sr=16000)

# Preprocess
inputs = processor.feature_extractor(audio, sampling_rate=16000, return_tensors="pt")

# Generate transcription
with torch.no_grad():
    predicted_ids = model.generate(inputs.input_features)

# Decode
transcription = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
print("Sanlish transcription:", transcription[0])

Intended Use

Direct Use

Transcribing Santali speech into Sanlish (custom Roman transliteration) for downstream NLP tasks.

Downstream Use

This model is designed to feed into a Machine Translation (MT) model (NLLB-200) for end-to-end Santali Speech → Bangla Text translation.

Out-of-Scope Use

Not suitable for general-purpose ASR in other languages
Not recommended for standard Santali Ol Chiki script transcription
Performance may degrade on speaker accents or recording conditions not represented in the 502-utterance training set

Limitations & Bias

Trained on only 502 utterances — a very small dataset by ASR standards
May not generalize well to unseen speakers or recording environments
Sanlish transliteration scheme has inherent inconsistencies (word boundary variation, hyphenation) which inflate WER beyond true phonetic error rate
No speaker diversity information available for the training set

Citation

If you use this model in your research, please cite:

bibtex
@misc{thunderboltc2025whisper-santali,
  author = {thunderboltc},
  title = {Whisper Small Fine-tuned on Santali Custom Transliteration (Sanlish)},
  year = {2025},
  publisher = {Hugging Face},
  url = {https://huggingface.co/thunderboltc/whisper-small-santali-sanlish}
}

Environmental Impact

Hardware: NVIDIA Tesla T4
Cloud Provider: Google Colab
Hours used: ~1.25 hours
Carbon emissions: Estimated using the ML Impact Calculator

whisper-small-santali-sanlish

README

Model Details

Model Description

What is Sanlish?

Training Details

Training Data

Training Procedure

Preprocessing

Training Hyperparameters

Best Checkpoint

Evaluation

Results on Test Set (51 utterances)

Sample Predictions

How to Get Started

Intended Use

Direct Use

Downstream Use

Out-of-Scope Use

Limitations & Bias

Citation

Environmental Impact

Explore FriendliAI today

README

Model Details

Model Description

What is Sanlish?

Training Details

Training Data

Training Procedure

Preprocessing

Training Hyperparameters

Best Checkpoint

Evaluation

Results on Test Set (51 utterances)

Sample Predictions

How to Get Started

Intended Use

Direct Use

Downstream Use

Out-of-Scope Use

Limitations & Bias

Citation

Environmental Impact