Results (932-clip held-out test set)
Table with columns: WER, CER | WER | CER |
|---|
| Base (Qwen3-ASR-0.6B-hf, zero-shot) | 0.756 | 0.454 |
| This model (fine-tuned) | 0.676 | 0.408 |
LLM-ASR (multimodal). Full fine-tune, targets stripped of tashkil + tags (trial t3). 3 epochs. Needs transformers>=5.14 for qwen3_asr.
Model comparison — all Arabic ASR models
Same 932-clip held-out test set, same clean_text scoring (strip tashkil + tags,
keep punctuation + dialect spelling) — so every row is directly comparable.
Table with columns: Model, Params, Zero-shot WER, Fine-tuned WER, CER (best)| Model | Params | Zero-shot WER | Fine-tuned WER | CER (best) |
|---|
| whisper-large-v3-turbo 🏆 | 809M | 0.590 | 0.344 | 0.115 |
| cohere-transcribe-arabic | 2.0B | 0.457 | 0.357 | 0.137 |
| whisper-medium | 769M | 0.717 | 0.358 | 0.123 |
| nemotron-3.5-asr (streaming) | 638M | 0.592 | 0.422 | — |
| whisper-small | 244M |
- Best fine-tuned:
whisper-large-v3-turbo (WER 0.344), with cohere-transcribe-arabic
a close second (0.357).
- Best zero-shot:
cohere-transcribe-arabic (0.457, Arabic-specialized). A full
fine-tune (all ~2B params, low LR) now improves it to 0.357; an earlier 32 GB
LoRA attempt had instead degraded it (0.510, overfit) — full-parameter tuning
with best-checkpoint selection was the fix.
- Streaming / low-latency:
nemotron-3.5-asr.
- Parakeet-TDT-0.6b-v3 was tried but abandoned (European-only pretraining;
cross-lingual transfer to Arabic converged far too slowly, WER ~0.93 after 3k steps).
Dataset
Fine-tuned on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private).
- ~40,000 train / 1,000 test clips, 16 kHz mono, single-channel.
- Multi-dialect Arabic: Levantine (Lebanese/Syrian), Maghrebi
(Moroccan/Algerian/Tunisian), Egyptian, Gulf/Saudi, Sudanese, Iraqi, and MSA.
- Augmented: each clean clip is expanded with variants (noise, music, speed,
voice, reverb/codec) via an
augmentation column. Derived from
oddadmix/dialectal-arabic-lahgtna-v2-smaller.
Preprocessing (applied to targets)
- Stripped tashkil/harakat (diacritics) and tatweel; removed non-verbal tags
(
[laughter], [exhale], [inhale], [mumble], [cough], timestamps, …).
- Kept dialectal consonants (گ ڨ چ پ ژ) — they encode real phonemes — and
sentence punctuation. Output is undiacritized.
- Filtered: dropped clips > 30 s (~10%; Whisper's encoder limit) and a few
corrupt rows (12k-token transcripts). ~62% of rows had harakat, ~28% had
non-verbal tags before cleaning. → 36,769 train / 932 test after filtering.
Training
- Base:
Qwen/Qwen3-ASR-0.6B-hf · full fine-tune (no LoRA)
- Effective batch 16 · LR 1e-5 (warmup 500, cosine) ·
6903 steps · best @ step None · ~102 min
- Hardware: a single 32 GB GPU, bf16 autocast
(fp32 weights + generate)
- Metric: WER/CER on cleaned references (clean-text normalized)
Fine-tuning (reproduce)
The exact fine-tuning code is bundled in this repo (train_qwen.py, normalize.py, eval_hf_asr.py) plus requirements.txt. See FINETUNE.md for the full walkthrough + lessons learned. Trained on oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented (private) — swap in any HF audio dataset with audio + text columns. normalize.py is the shared text cleaning (strip tashkil + non-verbal tags, keep dialectal letters گ ڨ چ).
pip install -r requirements.txt # recent transformers (qwen3_asr) + peft
python train_qwen.py --model_id Qwen/Qwen3-ASR-0.6B-hf \
--dataset_name oddadmix/dialectal-arabic-lahgtna-v2-smaller-augmented --audio_column audio --text_column text \
--eval_split test --language Arabic --strip_tashkil --strip_tags \
--output_dir my-run
python eval_hf_asr.py --model_type qwen --model my-run # WER / CER
Note: Qwen3-ASR's chat template drops the transcript during training — train_qwen.py
builds the target manually with the prompt masked to -100 (see its DataCollator).
Usage
import torch, torchaudio
from transformers import AutoProcessor, Qwen3ASRForConditionalGeneration
repo = "oddadmix/qwen3-asr-0.6b-arabic-dialectal"
proc = AutoProcessor.from_pretrained(repo)
model = Qwen3ASRForConditionalGeneration.from_pretrained(
repo, torch_dtype=torch.bfloat16).to("cuda").eval()
wav, sr = torchaudio.load("clip.wav")
inp = proc.apply_transcription_request(audio=wav.mean(0).numpy(),
language="Arabic").to(model.device, model.dtype)
ids = model.generate(**inp, max_new_tokens=256)
print(proc.batch_decode(ids, skip_special_tokens=True)[0])
Learnings & notes
- Data cleaning matters most: keeping dialectal letters + stripping tashkil/tags
(rather than aggressive letter-folding) was key for a multi-dialect model.
- Whisper fine-tunes furthest; the offline (full-context) models beat the
streaming Nemotron on pure WER, though Nemotron has the best zero-shot (native
Arabic) and offers streaming.
- Blackwell (sm_120) GPU gotchas for the NeMo/Nemotron model: numba's
RNNT loss needs
numba==0.61.2 + NUMBA_CUDA_USE_NVIDIA_BINDING=1 +
cuda-python<13 to emit sm_120 code, and fp32 (the numba loss can't take
bf16). Nemotron 3.5 requires NeMo main (its EncDecRNNTBPEModelWithPrompt
class isn't in any release) and a per-sample lang=ar prompt.
Limitations
- Trained on augmented, partly synthetic multi-dialect data; real-world dialect
coverage varies (Maghrebi is the hardest).
- Output is undiacritized and lower-cased for Latin tokens.
- Offline model (≤30 s chunks); not streaming.
