itayinbar

whisper-base-he

README

License: apache-2.0

Results

Word error rate, scored with a harness that reproduces the ivrit.ai Hebrew leaderboard on 40 of 40 published model and dataset pairs.

Table with columns: Benchmark, this model, 50.2M, whisper-small, 242M, whisper-base, 72.6M, ivrit.ai large-v3-turbo, 809M
Benchmark	this model, 50.2M	whisper-small, 242M	whisper-base, 72.6M	ivrit.ai large-v3-turbo, 809M
`ivrit-ai/eval-d1`	14.87%			5.5%
`imvladikon/hebrew_speech_kan`	18.47%	37.42%	68.12%	8.10%
`ivrit-ai/eval-whatsapp`	25.77%			6.1%
`google/fleurs` he	33.74%	45.94%	68.19%	18.72%

Against every open Hebrew model we could find, on FLEURS-he:

Table with columns: Model, Parameters, WER
Model	Parameters	WER
ivrit-ai/whisper-large-v3-turbo	809M	18.72%
whisper-base-he	50.2M	33.74%
openai/whisper-small	242M	45.94%
mike249/whisper-tiny-he-2	37.8M	66.98%
openai/whisper-base	72.6M	68.19%
Alex2575/aleksis_heb_base

It runs at 500 to 800 times realtime on an RTX 5070 Laptop: a 47-minute recording transcribes in 3.6 seconds.

Why it is smaller than the model it came from

Whisper's vocabulary holds 51,865 tokens for 99 languages. On 24,225 words of held-out Hebrew speech it spends 3.17 tokens per Hebrew word, against 1.28 for English, and its embedding table is 26.6M of whisper-base's 72.6M parameters, nearly all of it tokens Hebrew never emits.

An 8,192-token Hebrew byte-level BPE brings that to 1.76 tokens per word and the model to 50.23M parameters, 30.8% smaller. Every new embedding row is initialised from the pretrained table, averaging the old rows for the same text, so none start from noise.

Measured on identical data and schedule, that vocabulary is worth 7.5 WER points on its own, not merely the size saving.

Usage

python
from transformers import AutoModelForSpeechSeq2Seq, AutoTokenizer, AutoFeatureExtractor
from transformers.generation.utils import GenerationMixin

model = AutoModelForSpeechSeq2Seq.from_pretrained("itayinbar/whisper-base-he")
tokenizer = AutoTokenizer.from_pretrained("itayinbar/whisper-base-he")
features = AutoFeatureExtractor.from_pretrained("itayinbar/whisper-base-he")

inputs = features(audio_16khz, sampling_rate=16000,
                  return_tensors="pt", padding="max_length")
# Monolingual: no language token, no task token, no timestamps. Whisper's own
# generate() expects all three, so call the generic implementation.
ids = GenerationMixin.generate(model, **inputs, max_new_tokens=200)
print(tokenizer.decode(ids[0], skip_special_tokens=True))

Three things differ from stock Whisper:

Call the generic generate. This model has no language or task tokens.
Pad mel features to the full 30-second window, as Whisper always requires.
Cut audio longer than 30 seconds into overlapping windows and stitch the results. Without that, eval-d1 scores 91.94% instead of 14.87%, because everything past the first thirty seconds is scored as deleted.

ONNX weights are included for transformers.js. A browser downloads 160.2 MB (fp16 encoder plus fp32 decoder), against 563 MB for the large Hebrew model it is meant to replace.

Only those two weight files are shipped. An fp16 decoder and an int8 encoder were built and then removed: the fp16 decoder fails to load in ONNX Runtime, and the int8 encoder loads but changes the transcript. Every combination offered here was checked by transcribing with it and comparing against the fp32 output, not by loading it.

Training

3,112.7 hours over 552,327 utterances, all human-transcribed.

Table with columns: Corpus, Hours
Corpus	Hours
`ivrit-ai/knesset-plenums-whisper-training`	2,756.6
`ivrit-ai/crowd-transcribe-v5`	295.0
`ivrit-ai/crowd-recital-whisper-training`	44.7
`google/fleurs` he train	9.5
`imvladikon/hebrew_speech_kan` train	6.9

Schedule-free AdamW, learning rate 1e-4, batch 16, bf16, 103,551 steps on a single RTX 5070 Laptop GPU with 8 GB. Corpora are sampled by target share of audio time rather than of utterances, since Knesset ships 30-second windows and crowd-transcribe averages 5.2 seconds.

What was tried and did not work

Measured on identical data and budget, so the comparisons are like for like:

Table with columns: Change, Effect on WER
Change	Effect on WER
Hebrew tokenizer instead of Whisper's	-7.5 points, and 30.8% fewer parameters
700 hours to 3,113 hours	-2.7 points on average
SpecAugment	none
Hybrid CTC objective	none, at 19% lower throughput

SpecAugment and hybrid CTC are standard practice in speech recognition, and neither did anything here.

Licence and provenance

Weights are Apache-2.0, following openai/whisper-base.

Training data from ivrit.ai under the ivrit.ai licence, which permits training models including commercially and requires attribution. Credit for that data belongs to ivrit.ai, whose own Hebrew models are the state of the art this one is measured against. FLEURS is CC-BY-4.0.

imvladikon/hebrew_speech_kan declares no licence on the Hub. It contributed 6.9 of 3,112.7 hours and roughly 4% of the audio the model heard after weighting. It is named here so anyone relying on this model's provenance can judge that themselves.

Code, evaluation harness and full method: github.com/itayinbarr/Hebrew-small-asr.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

itayinbar

Model Tree

Base

openai/whisper-base

Quantized

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer