VoicenterTeam

whisper-heb-v8-multi

README

License: apache-2.0

Why v8

On telephony audio Whisper fine-tunes hallucinate short phrases when there is no speech. v7 produced a transcript for 98% of real line-noise clips (27% of them "תודה רבה") and 46% of synthetic noise. v8 was warm-started from v7 and trained 5,000 steps on the v7 corpus plus 83,920 speech-free rows (real Voicenter line noise ×4, synthetic noise/music/tones, real speech with trailing noise) whose target is Whisper's native <|nospeech|> token, with the 458 "phantom thanks" training rows and ~2.6k mis-cut labels removed.

Table with columns: speech-free set, v7 any output, v8 any output, v7 "תודה", v8 "תודה"
speech-free set	v7 any output	v8 any output	v7 "תודה"	v8 "תודה"
real Voicenter line noise (59 clips, human-labelled, held out)	98.3%	6.8%	27.1%	0.0%
synthetic noise, 12 kinds (1,500 clips)	46.2%	1.3%	8.5%	0.0%

Speech is unaffected: on 1,517 real telephony clips ≤4 s (56 of them a bare "תודה רבה") v8 scores WER 0.033 vs v7's 0.034 and silences 0.4% of them.

Usage

python

from transformers import pipelineasr = pipeline("automatic-speech-recognition", model="VoicenterTeam/whisper-heb-v8-multi",               torch_dtype="bfloat16", device="cuda")asr("call.wav")                                        # language auto-detected; "" on noiseasr("call.wav", generate_kwargs={"language": "he"})    # force one of he/en/ar/ru/yi

The stock generation_config is kept (begin_suppress_tokens=[220, 50257], language=None). On speech-free input the model emits <|nospeech|> followed by <|endoftext|>, which decodes to an empty string; in faster-whisper / CTranslate2 this surfaces as a high no_speech_prob, so the usual no_speech_threshold drops the segment. Converts with ct2-transformers-converter unchanged.

Results (clean audio, auto language detection, WER)

Normalised text (nikud/tashkeel/punctuation stripped, case folded, ё→е, Arabic alef/ta marbuta/ya folded), greedy, bf16. Full tables incl. 8 kHz μ-law condition, forced mode, CER, LID confusion: results/whisper-heb-v8-multi/SUMMARY.md in the training repo.

Table with columns: lang, sets, v8, v7, openai large-v3-turbo, LID acc
lang	sets	v8	v7	openai large-v3-turbo	LID acc
he	ILSpeech, 10k telephony	0.128	0.128	0.240	1.00
en	FLEURS+CV+AMI, AppTek call-center	0.141	0.138	0.133	0.98
ar	FLEURS, CV17, Casablanca jo/ps

Selected sets:

Table with columns: set, refs, v8, v7
set	refs	v8	v7
he telephony 10k (1000 sample)	soniox	0.155	0.155
he telephony, soniox+witness agree (5,915)	soniox	0.078	0.079
he short clips ≤4 s (1,517)	2-provider	0.033	0.034
he ILSpeech speaker 1	human	0.124	0.124

Regressions to know about: Russian real phone calls in auto-LID mode (0.412 vs 0.388; identical when the language is forced — a few more calls are misdetected), and the Hebrew kan set in forced mode (0.088 vs 0.073; equal in auto mode). Everything else is equal or better.

Known limitations

Arabic trails openai/whisper-large-v3-turbo (0.368 vs 0.292): Arabic was 5% of the training mix with a Levantine-biased teacher. Russian telephony also trails on real calls.
LID drops on very short clips (<2 s: ar 0.58, yi 0.83, ru 0.89) and ~5% of Hebrew read speech (FLEURS, CV) is detected as Arabic — force language="he" when the language is known.
About 7% of real line-noise clips still get a short output (mostly English fillers). A VAD in front of the model remains cheap insurance.
Yiddish numbers are read speech; no conversational Yiddish test set exists.

Training

Warm start: VoicenterTeam/whisper-heb-v7-multi (step 14500). 5,000 steps, effective batch 256, lr 1.5e-6 (encoder 0.3×), cosine, 200 warmup, bf16, label smoothing 0.1, SpecAugment, speed perturbation, telephone band-pass. Best checkpoint (step 3500) selected on macro dev WER over the five languages with auto language detection.
Data: v7 corpus (3.76M clips; he 53%, en 19%, ru 19%, ar 5%, yi 3%, commercially-licensed sources only) + 83,920 speech-free rows with target <|nospeech|><|endoftext|> − 458 phantom "תודה רבה" rows − ~2.6k decoder-loop / mis-cut / subtitle-credit labels.
Knowledge distillation as in v7 (0.2·CE + 0.8·T²·KL, T=2; teachers he: whisper-heb-v5, en: TheStageAI/thewhisper-large-v3-turbo, ru: openai/whisper-large-v3, ar: VoicenterTeam/whisper-levantine-hf, yi: ivrit-ai/yi-whisper-large-v3). Negatives are CE-only: every teacher hallucinates on noise, so distilling them there would teach the phantom back.

Code: the whisper-finetune repo (scripts/build_multilingual_manifest_v8.py, plans/v8-data/NOTES.md, src/kd.py, scripts/bench_v7.sh).

Data attribution

As for v7: ivrit.ai (crowd-recital, crowd-transcribe, audio-v2, crowd-recital-yi, crowd-whatsapp-yi; ivrit.ai license v2 — not for voice cloning), VoxKnesset, Voicenter telephony; Common Voice (CC0), People's Speech, Granary/YODAS, VoxPopuli, AMI, ICSI, LibriSpeech, EdAcc, NOTSOFAR-1, DiPCo, SLUE, Vystadial, MLS, speechocean762, MInDS-14; Golos (SberDevices license), Russian LibriSpeech, SOVA RuDevices, FLEURS, ToneWebinars, Dialogs; Omnilingual ASR corpus (CC-BY-4.0), Lahgtna Levantine TTS, SADA, ClArTTS, ArVoice, SVQ; REYD (CC-BY-4.0). Negatives: Voicenter line noise, MUSAN (CC-BY-4.0) noise and music. Teachers: TheStage AI (CC-BY-4.0), OpenAI Whisper (MIT), ivrit.ai (Apache-2.0).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

VoicenterTeam

Model Tree

Base

VoicenterTeam/whisper-heb-v7-multi

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer