oddadmix

whisper-large-v3-turbo-arabic-dialectal

README

License: apache-2.0

Results (932-clip held-out test set)

Table with columns: WER, CER
	WER	CER
Base (whisper-large-v3-turbo, zero-shot)	0.590	0.278
This model (fine-tuned)	0.344	0.115

BEST model. Ships in fp16 so training force-loads fp32 (fp16 weights crash generate() at eval). Extending to 12k steps overfit (WER stalled ~0.35, eval_loss rose) — 6k is the sweet spot here.

Model comparison — all Arabic ASR models

Same 932-clip held-out test set, same clean_text scoring (strip tashkil + tags, keep punctuation + dialect spelling) — so every row is directly comparable.

Table with columns: Model, Params, Zero-shot WER, Fine-tuned WER, CER (best)
Model	Params	Zero-shot WER	Fine-tuned WER	CER (best)
whisper-large-v3-turbo 🏆	809M	0.590	0.344	0.115
cohere-transcribe-arabic	2.0B	0.457	0.357	0.137
whisper-medium	769M	0.717	0.358	0.123
nemotron-3.5-asr (streaming)	638M	0.592	0.422	—
whisper-small	244M

Best fine-tuned: whisper-large-v3-turbo (WER 0.344), with cohere-transcribe-arabic a close second (0.357).
Best zero-shot: cohere-transcribe-arabic (0.457, Arabic-specialized). A full fine-tune (all ~2B params, low LR) now improves it to 0.357; an earlier 32 GB LoRA attempt had instead degraded it (0.510, overfit) — full-parameter tuning with best-checkpoint selection was the fix.
Streaming / low-latency: nemotron-3.5-asr.
Parakeet-TDT-0.6b-v3 was tried but abandoned (European-only pretraining; cross-lingual transfer to Arabic converged far too slowly, WER ~0.93 after 3k steps).

Dataset

Fine-tuned on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private).

~40,000 train / 1,000 test clips, 16 kHz mono, single-channel.
Multi-dialect Arabic: Levantine (Lebanese/Syrian), Maghrebi (Moroccan/Algerian/Tunisian), Egyptian, Gulf/Saudi, Sudanese, Iraqi, and MSA.
Augmented: each clean clip is expanded with variants (noise, music, speed, voice, reverb/codec) via an augmentation column. Derived from oddadmix/dialectal-arabic-lahgtna-v2-smaller.

Preprocessing (applied to targets)

Stripped tashkil/harakat (diacritics) and tatweel; removed non-verbal tags ([laughter], [exhale], [inhale], [mumble], [cough], timestamps, …).
Kept dialectal consonants (گ ڨ چ پ ژ) — they encode real phonemes — and sentence punctuation. Output is undiacritized.
Filtered: dropped clips > 30 s (~10%; Whisper's encoder limit) and a few corrupt rows (12k-token transcripts). ~62% of rows had harakat, ~28% had non-verbal tags before cleaning. → 36,769 train / 932 test after filtering.

Training

Base: openai/whisper-large-v3-turbo · full fine-tune (no LoRA)
Effective batch 32 · LR 1e-5 (warmup 500, cosine) · 6000 steps · best @ step 6000 · ~200 min
Hardware: a single 32 GB GPU, bf16 autocast (fp32 weights + generate)
Metric: WER/CER on cleaned references (clean-text normalized)

Fine-tuning (reproduce)

The exact fine-tuning code is bundled in this repo (train.py, normalize.py, evaluate_model.py) plus requirements.txt. See FINETUNE.md for the full walkthrough + lessons learned. Trained on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private) — swap in any HF audio dataset with audio + text columns. normalize.py is the shared text cleaning (strip tashkil + non-verbal tags, keep dialectal letters گ ڨ چ).

bash
pip install -r requirements.txt
huggingface-cli login          # for the (private) dataset

python train.py \
  --base_model openai/whisper-large-v3-turbo --run_name my-run \
  --per_device_train_batch_size 8 --gradient_accumulation_steps 4 \
  --learning_rate 1e-5 --warmup_steps 500 --max_steps 6000

python evaluate_model.py --model runs/my-run          # WER / CER

Note: some checkpoints (e.g. large-v3-turbo) ship in fp16 — train.py force-loads fp32 so generate() doesn't crash at eval. bf16 autocast is used for training.

Usage

python
import torch, torchaudio
from transformers import WhisperForConditionalGeneration, WhisperProcessor

repo = "oddadmix/whisper-large-v3-turbo-arabic-dialectal"
proc = WhisperProcessor.from_pretrained(repo)
model = WhisperForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.float16).to("cuda").eval()

wav, sr = torchaudio.load("clip.wav")          # resample to 16 kHz mono if needed
feats = proc(wav.mean(0).numpy(), sampling_rate=16000,
             return_tensors="pt").input_features.to("cuda", torch.float16)
ids = model.generate(feats, language="ar", task="transcribe")
print(proc.batch_decode(ids, skip_special_tokens=True)[0])

Learnings & notes

Data cleaning matters most: keeping dialectal letters + stripping tashkil/tags (rather than aggressive letter-folding) was key for a multi-dialect model.
Whisper fine-tunes furthest; the offline (full-context) models beat the streaming Nemotron on pure WER, though Nemotron has the best zero-shot (native Arabic) and offers streaming.
Blackwell (sm_120) GPU gotchas for the NeMo/Nemotron model: numba's RNNT loss needs numba==0.61.2 + NUMBA_CUDA_USE_NVIDIA_BINDING=1 + cuda-python<13 to emit sm_120 code, and fp32 (the numba loss can't take bf16). Nemotron 3.5 requires NeMo main (its EncDecRNNTBPEModelWithPrompt class isn't in any release) and a per-sample lang=ar prompt.

Limitations

Trained on augmented, partly synthetic multi-dialect data; real-world dialect coverage varies (Maghrebi is the hardest).
Output is undiacritized and lower-cased for Latin tokens.
Offline model (≤30 s chunks); not streaming.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

oddadmix

Model Tree

Base

openai/whisper-large-v3-turbo

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality