VoicenterTeam

whisper-heb-v7-multi

README

License: apache-2.0

Usage

python

from transformers import pipelineasr = pipeline("automatic-speech-recognition", model="VoicenterTeam/whisper-heb-v7-multi",               torch_dtype="bfloat16", device="cuda")asr("call.wav")                                              # language auto-detectedasr("call.wav", generate_kwargs={"language": "he"})          # or force one of he/en/ar/ru/yi

The shipped generation_config has language=None, so plain generate() runs Whisper's language detection with the fine-tuned weights. Forcing the language gives slightly better WER on short Arabic/Russian clips (see below) and is what to do when the language is known. The model is a stock Whisper checkpoint: it converts to CTranslate2 / faster-whisper with ct2-transformers-converter unchanged.

Results

WER on normalized text (nikud/tashkeel/punctuation stripped, case folded, ё→е, Arabic alef/ta marbuta/ya folded), greedy decoding, bf16. auto = language detected by the model. Compared with our previous Hebrew-only model (v6) and openai/whisper-large-v3-turbo. Full tables incl. 8 kHz simulated-telephony condition, forced-language mode, CER and per-set flags: results/whisper-heb-v7-multi/SUMMARY.md in the training repo.

Per language (macro over the language's test sets, clean audio, auto LID)

Table with columns: lang, sets, v7, v6 (Hebrew-only), openai turbo, v7 LID acc
lang	sets	v7	v6 (Hebrew-only)	openai turbo	v7 LID acc
he	ILSpeech, 10k telephony	0.128	0.134	0.240	1.00
en	FLEURS+CV+AMI, AppTek call-center	0.138	1.067	0.133	0.99
ar	FLEURS, CV17, Casablanca jo/ps

Under a simulated 8 kHz μ-law phone channel the macro is 0.239 (v6 0.867, openai 0.430).

Selected sets

Table with columns: set, refs, v7, v6, openai turbo
set	refs	v7	v6	openai turbo
he telephony, 10k (1000 sample)	soniox	0.155	0.166	0.342
he telephony, soniox+witness agree ≤0.2 WER (5,915)	soniox	0.079	0.088	0.221
he ILSpeech speaker 1 (579)	human	0.126	0.125	0.148

What this means

Hebrew telephony improved over the Hebrew-only v6 (10k: 0.166 → 0.155; agreement subset 0.088 → 0.079); long-form whatsapp and ILSpeech are unchanged within noise. Hebrew read speech regressed a little (FLEURS 0.180 → 0.204, Common Voice 0.134 → 0.150, forced-he).
English and Russian are at openai-turbo level (within ±0.01), from WER ≈ 1.0 in v6, which had lost them entirely during Hebrew fine-tuning.
Yiddish (Hebrew script, Hasidic orthography) works: 0.35 WER where openai turbo writes Latin-script German-ish text (1.2).
Arabic is the weak spot: 0.376 vs 0.292 for openai turbo. Arabic was 5% of the training mix with a Levantine-biased teacher; the test set mixes MSA and several dialects.
Russian telephony also lags openai turbo (0.388 vs 0.316 on real calls): the commercial Russian training data has no telephone speech.
Language ID: 0.99 overall; Hebrew is never misdetected on telephony. On very short clips (<2 s) LID drops (ar 0.58, yi 0.83, ru 0.89), and on Hebrew read speech (FLEURS, CV) ~5% of clips are detected as Arabic — force language="he" when the language is known. He↔Yi confusion is 4 of 1,104 Yiddish clips and 0 of 7,875 Hebrew clips.

Known limitations

Hallucinates short phrases on speech-free audio (line noise, music, hold tones): 98% of real line-noise clips get a transcript, 27% of them "תודה רבה". A follow-up (v8) trains explicit speech-free negatives; until then gate the model with a VAD.
No conversational Yiddish test set exists; the Yiddish numbers are read speech.
Hebrew telephony references are soniox transcripts (agreement, not ground truth); the human-referenced Hebrew sets are the ivrit.ai suite above.
No code-switching (he+en in one utterance) evaluation.

Training

Data: 3.76M clips. Hebrew 53% (Voicenter 8 kHz telephony pseudo-labelled by soniox, ivrit.ai audio-v2 podcasts, Knesset, crowd recordings, synthetic TTS), English 19%, Russian 19%, Arabic 5%, Yiddish 3% (3× oversampled). Only commercially-licensed sources for en/ru/ar/yi. Labels keep case and punctuation where the source had it; Hebrew/Yiddish nikud stripped.
Label format: <|startoftranscript|><|lang|><|transcribe|><|notimestamps|> text <|endoftext|> with the row's own language token, so the language token is the first predicted token.
Knowledge distillation: loss = 0.2·CE + 0.8·T²·KL(teacher‖student), T=2, one frozen teacher per language sharing the Whisper large-v3 tokenizer — he: whisper-heb-v5 (ours), en: TheStageAI/thewhisper-large-v3-turbo, ru: openai/whisper-large-v3, ar: VoicenterTeam/whisper-levantine-hf, yi: ivrit-ai/yi-whisper-large-v3. The language token position is excluded from the KL.
Optimisation: lr 3e-6 (encoder 0.3×), cosine, 500 warmup, effective batch 256, bf16, label smoothing 0.1, SpecAugment, speed perturbation 0.9–1.1×, telephone band-pass augmentation.
Checkpoint selection on multi_wer: macro WER over 150-clip dev slices per language (sentence-disjoint from training) + ILSpeech speaker 1, auto language detection.

Code: the whisper-finetune repo (scripts/launch_multilingual_training.sh, src/kd.py, scripts/bench_v7.sh).

Data attribution

Hebrew: ivrit.ai (crowd-recital, crowd-transcribe, audio-v2; ivrit.ai license v2 — not for voice cloning), VoxKnesset, Voicenter telephony. English: Common Voice 17 (CC0), People's Speech (CC-BY), Granary/YODAS (CC-BY), VoxPopuli (CC0), AMI & ICSI (CC-BY-4.0), LibriSpeech (CC-BY-4.0), EdAcc (CC-BY-SA), NOTSOFAR-1 (CC-BY-4.0), DiPCo (CDLA-P), SLUE-HVB/VoxCeleb (CC-BY-4.0), Vystadial (CC-BY-SA-3.0), MLS (CC-BY-4.0), speechocean762 (CC-BY-4.0), MInDS-14 (CC-BY-4.0). Russian: Golos (SberDevices license), YODAS/Granary (CC-BY), Russian LibriSpeech (CC-BY-4.0), SOVA RuDevices (CC-BY-4.0), FLEURS (CC-BY-4.0), Common Voice (CC0), ToneWebinars (MIT), Dialogs (OpenRAIL). Arabic: Common Voice 17 (CC0), Omnilingual ASR corpus (CC-BY-4.0), Lahgtna Levantine TTS (CC-BY-4.0, synthetic), SADA, ClArTTS, ArVoice, FLEURS, SVQ, YODAS. Yiddish: ivrit.ai crowd-recital-yi / crowd-whatsapp-yi, Omnilingual ASR corpus ydd (CC-BY-4.0), REYD (CC-BY-4.0), Common Voice 22 (CC0). Teachers: TheStage AI (CC-BY-4.0), OpenAI Whisper (MIT), ivrit.ai (Apache-2.0).

Evaluation data notes

ILSpeech speaker 2 turned out to be Making History podcast audio that is also in the ivrit.ai audio-v2 training corpus; all ILSpeech numbers here use speaker 1 only (the full-set figure is contaminated for v5/v6/v7). Casablanca and Common Voice test sentences appear in some public Arabic fine-tunes' training data; none of ours.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

VoicenterTeam

Model Tree

Base

ivrit-ai/whisper-large-v3-turbo

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer