coriollon

whisper-large-v3-turbo-russian

README

License: apache-2.0

Headline metrics (6 RU test sets, N=500 each)

Table with columns: Metric, Vanilla turbo, Stage AI (prev), Stage AS v1, Stage AS v2 (this)
Metric	Vanilla turbo	Stage AI (prev)	Stage AS v1	Stage AS v2 (this)
WER (aggregate)	13.25 %	9.43 %	9.83 %	8.93 %
PER F1 (punct macro F1)	≈ 0.05	0.20	0.39	0.41
Cap F1 (word-level case-match)	≈ 0.20	0.48	0.83	0.83
Sent F1 (sentence-boundary F1)	≈ 0.15	0.45	0.81	0.80

v2 vs v1: WER −0.90 pp, PER +0.02, Cap ~stable, Sent ~stable.

Metric correction (2026-07-19). The per-mark F1 table below was regenerated after fixing a bug in the evaluation harness that ignored leading punctuation on tokens (Russian dialogue em-dash — слово… and standalone ...). The trained model was actually emitting these marks all along; only the metric didn't see them. Aggregate WER / Cap / Sent are unaffected. The correct em-dash F1 for v2 is 0.322, not the 0.000 previously reported.

Per-dataset delta (Stage AS v1 → v2)

Table with columns: Dataset, WER%, PER F1, Cap F1, Sent F1
Dataset	WER%	PER F1	Cap F1	Sent F1
Common Voice 21 RU	−0.5	+0.00	+0.01	+0.01
RuLibriSpeech	−0.9	+0.02	+0.03	+0.02
Sberdevices Golos far-field	−2.0	+0.05	+0.01	+0.01

Per-mark F1 (v2, corrected metric)

Table with columns: Mark, v2, non-zero on datasets
Mark	v2	non-zero on datasets
. period	0.759	6 / 6
, comma	0.734	6 / 6
? question	0.770	6 / 6
! exclamation	0.381	5 / 6
— dash	0.322	6 / 6
… ellipsis	0.038

Dash performance is strongest on Common Voice (F1 0.612) and degrades on the noisier Golos / SOVA sources. Ellipsis remains hard to measure — only SOVA test refs contain gold ellipsis, and only 29 rows across ~2 400 aligned test samples. quote_open/quote_close low scores are chiefly a reference-set artifact: quotes were only present in the augmented SOVA and Golos refs and inconsistently formatted.

Real-world qualitative check: dialogue lines from Russian audiobooks and podcasts produce dashes on speaker turns; short quoted phrases in news / interview audio produce guillemets. The metric here understates what the model actually does on well-formatted references.

What was fixed

The previous Stage AI release used GigaAM pseudo-labels for the hardtail samples in training. GigaAM emits mostly periods and sentence-initial capitals — Stage AI inherited that poverty. Around 46 % of the training-label corpus had zero punctuation by construction.

Stage AS v1: re-labelled the punct-poor slice via an offline LLM pass (Gemma 3 12B, WER ≤ 1 % word-preservation verification), yielding a 108 K-row publication-quality training corpus. LoRA fine-tune (r=32, q/k/v/o + fc1/fc2, 2 epochs, lr 5e-6).

Stage AS v2 (this release): larger LoRA (r=48, 42 M trainable params), 3 epochs, WeightedRandomSampler ×2 on rows containing rare marks (—, –, …, !, ?) so 32 % of samples surface at double frequency. WER dropped nearly a point aggregate — including −2 pp on both Golos splits.

Evaluation methodology

PER macro F1 — per-mark F1 (period, comma, question, exclamation, em-dash, ellipsis, quotes) averaged across mark types that appear at least once in the reference.
Cap F1 — per-word case-match F1 over words that survive ref↔hyp alignment.
Sent F1 — sentence-boundary F1 based on [.!?…] positions, 0.02-fraction tolerance for matching.
WER — standard, computed on lowercase punct-stripped text.

Golos and SOVA test-set references lack punctuation — measuring punct/cap F1 against them without augmentation always yields 0 for both models. The aggregate row above uses augmented references from Gemma 3 12B with the same word-preservation verification as training (1256 augmented / 234 rejected / 0 errors over Golos+SOVA). Both v1 and v2 evaluated on the same augmented references. Un-augmented CV21 / RuLibri / Podlodka refs are kept as-is.

Usage

python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
import torch

processor = WhisperProcessor.from_pretrained("coriollon/whisper-large-v3-turbo-russian")
model = WhisperForConditionalGeneration.from_pretrained(
    "coriollon/whisper-large-v3-turbo-russian",
    torch_dtype=torch.float16,
).to("cuda")

inputs = processor(audio_array, sampling_rate=16000, return_tensors="pt")
input_features = inputs.input_features.to("cuda", dtype=torch.float16)
predicted_ids = model.generate(input_features, language="ru", task="transcribe")
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]

Pre-quantized faster-whisper variants

Table with columns: Folder, Quantization, Size
Folder	Quantization	Size
`ct2_int8_float16/`	int8 weights + fp16 compute	782 MB
`ct2-int16/`	int16 (Stage AS v1 build; rebuild pending)	1.6 GB

python
from huggingface_hub import snapshot_download
from faster_whisper import WhisperModel

ct2_path = snapshot_download(
    repo_id="coriollon/whisper-large-v3-turbo-russian",
    allow_patterns="ct2_int8_float16/*",
)
model = WhisperModel(f"{ct2_path}/ct2_int8_float16",
                    device="cuda", compute_type="int8_float16")
segments, _ = model.transcribe("audio.wav", language="ru", beam_size=5)

Limitations

English-only audio degraded vs vanilla turbo (Russian specialization tradeoff — inherited from Stage AI).
For pure-Russian deployments with no punctuation or case requirement, dedicated Russian ASR models like GigaAM outperform this on raw WER.
Ellipsis (…) F1 stays low: only SOVA test refs contain any gold ellipsis; the measured 0.038 is not a confident estimate.
On the un-augmented CV21 / RuLibri / Podlodka references the punct gains over vanilla turbo are smaller than the augmented-refs aggregate suggests; they're still positive.

License

Apache 2.0 (inherited from base Whisper model).

Changelog

Metric fix (2026-07-19) — Evaluation harness now counts leading em-dash (Russian dialogue) and standalone ... / … tokens. No change to the released weights, only to how their punctuation quality is reported. Corrected numbers: WER 8.93 %, PER 0.41, dash F1 0.322 (was reported as 0.000).
Stage AS v2 (2026-07-14) — LoRA r=48, 3 epochs, rare-mark ×2 sample weighting. WER 9.83 → 8.93 %. PER 0.39 → 0.41, Cap and Sent essentially preserved. Golos WER drops −2 pp on both splits.
Stage AS v1 (2026-07-12) — punctuation & capitalization LoRA fine-tune. Cap F1 0.48 → 0.83, Sent F1 0.45 → 0.81, PER F1 0.20 → 0.39. WER 9.43 → 9.83 %.
Stage AI (previous) — GigaAM-teacher relabel, WER 13.25 → 9.43 %.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

coriollon

Model Tree

Base

openai/whisper-large-v3-turbo

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer