Blgn94

whisper-small-mn-v3

README

License: apache-2.0

Results

All numbers are Word/Character Error Rate (lower is better). Normalized = lowercased, punctuation stripped, NFC-normalized Cyrillic (ё/й) — this is what published benchmarks use, so compare against the _norm columns.

Three evaluation sets are reported:

Table with columns: eval set, what it is, trust
eval set	what it is	trust
clean	dev clips whose transcripts never appear in training (241 clips)	the honest metric — model was selected on this
indomain	old random split; ~89% of its transcripts also appear in training (300 clips)	optimistic (memorization), kept for continuity
mic	10 clips recorded on a laptop mic — real, noisy, out-of-distribution	reported only, too small to trust

v25 final metrics

Table with columns: eval set, WER, CER, WER (norm), CER (norm)
eval set	WER	CER	WER (norm)	CER (norm)
clean	21.33	9.78	19.67	9.13
indomain	23.01	8.40	21.04	7.92
mic	84.36	48.66	72.51	44.87

Per-source normalized WER on the clean set: mbspeech 6.03, fleurs 21.67, youtube 57.65 (spontaneous speech is the weak spot — see Limitations).

A/B vs baseline (the reason this recipe was chosen)

Both runs share the identical text-disjoint clean eval set, so the comparison is fair.

Table with columns: metric (normalized), baseline, v25, Δ
metric (normalized)	baseline	v25	Δ
clean WER (selection metric)	22.85	19.67	−3.18
clean CER	9.82	9.13	−0.69
clean — mbspeech	8.18	6.03	−2.14
clean — youtube (spontaneous)	66.15	57.65

The baseline is a plain fine-tune on the full corpus with no de-duplication or augmentation. v25 improves the honest clean WER by 3.2 points and generalizes markedly better on spontaneous YouTube speech and on real microphone recordings. The in-domain score rises — this is expected and desirable: capping duplicate transcripts removes the memorization crutch (the corpus repeats each Common Voice sentence ~14.5×), so the optimistic in-domain number falls while genuine transcription of unseen speech improves.

Training data

Blgn94/mongolian-stt-dataset-v24 — ~148 h of 16 kHz Mongolian Cyrillic speech across four sources (measured by decoding all clips):

Table with columns: source, ~hours, notes
source	~hours	notes
Common Voice	~126	read speech; only ~6,000 distinct sentences (~14.5× text reuse)
FLEURS	~14	read speech
MBSpeech	~7	read speech
YouTube	~1	manual-subtitle clips — the only spontaneous speech

Known weakness: the corpus is ~85% Common Voice read speech and very light on spontaneous speech (~1 h). Read-speech sources score in the single-to-low-twenties WER; spontaneous YouTube sits near 58%. Expect the model to be strongest on clear, read-style Mongolian and weaker on conversational/noisy audio.

v25 corpus shaping (what makes this run different)

Per-sentence cap = 4: at most 4 recordings per distinct transcript, dropping 66.5% of rows (95,946 → 32,134) and cutting Common Voice text reuse from 14.5× to 4×. Breaks the transcript-memorization shortcut.
Source-balanced sampling (α = 0.5): re-weights so Common Voice does not drown out FLEURS, MBSpeech, and YouTube.
Audio augmentation (train only; eval always clean):
- SpecAugment (time masking, p = 0.05)
- Waveform augmentation (p = 0.5, up to 2 ops): speed 0.9–1.1×, gain, additive noise at 10–30 dB SNR, light reverb.

Training procedure

Table

Base model	`openai/whisper-small`
Precision	bf16
Effective batch	16 (batch 4 × grad-accum 4)
Learning rate	1e-5, linear schedule, warmup ratio 0.05
Weight decay	0.01
Label smoothing	0.0
Epochs	4 (8,036 steps)
Max label tokens	440
Seed

Model selection: best checkpoint by eval_clean_wer_norm (load_best_model_at_end). Labels drop the leading <|startoftranscript|> token (the model re-prepends decoder_start_token_id).

Reproduce

bash
MAX_PER_SENTENCE=4 SOURCE_ALPHA=0.5 AUG_PROB=0.5 EPOCHS=4 OUT=out_v25 \
  python scripts/train.py

Full config and final metrics are in run_manifest.json in this repo.

Usage

python
import torch
from transformers import pipeline

pipe = pipeline(
    "automatic-speech-recognition",
    model="Blgn94/whisper-small-mn-v25",
    device=0 if torch.cuda.is_available() else -1,
)
result = pipe(
    "audio.mp3",
    generate_kwargs={"language": "mn", "task": "transcribe"},
)
print(result["text"])

For long audio, add chunk_length_s=30. For raw model + processor access:

python
from transformers import WhisperForConditionalGeneration, WhisperProcessor

processor = WhisperProcessor.from_pretrained("Blgn94/whisper-small-mn-v25")
model = WhisperForConditionalGeneration.from_pretrained("Blgn94/whisper-small-mn-v25")

Limitations

Spontaneous / conversational speech: weak (~58% WER on YouTube), because the corpus has only ~1 h of it. Real-world microphone input is also harder (mic set ~72% WER, but only 10 clips — treat as directional, not a benchmark).
Not evaluated on the official Common Voice test split. The clean set is a custom text-disjoint split of v24; it is not speaker-disjoint from training, so real-world WER on entirely new speakers may be somewhat higher. For a speaker-disjoint reference point, bayartsogt/whisper-small-mn-8 reports ~26.5% WER on the official Common Voice mn test.
Cyrillic Khalkha only. No traditional Mongolian script; not tuned for other dialects.

License

Apache-2.0, inheriting from the openai/whisper-small base model. Note the underlying training data combines sources under their own licenses (Common Voice, FLEURS, MBSpeech, YouTube manual subtitles); review those before commercial use.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Blgn94

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer