TheBiggyBang

whisper-small-tunisian-derja

README

License: apache-2.0

Results

Held-out test set: 482 clips (36.6 min) drawn from all LinTO test splits. No recording in it was seen during training or checkpoint selection.

Table with columns: Model / decoder, WER, normalized WER (95 % CI), CER
Model / decoder	WER	normalized WER (95 % CI)	CER
whisper-small (no fine-tuning), CT2 beam 3	79.1	76.3 (73.8–79.0)	44.5
this model, Transformers fp16 greedy	32.8	31.9 (29.9–34.2)	13.4
this model, CTranslate2 int8 beam 3	30.4	29.5 (27.5–31.6)	12.0
LinTO Kaldi/Vosk (with 4-gram LM, scored on 400 of the 482 clips)	25.4	25.0 (22.0–28.1)	15.7

Normalized WER removes Derja spelling variants that are not recognition errors: أ/إ/آ→ا, ى→ي, ة→ه, Arabic-Indic digits, punctuation, lowercase Latin, and standalone «و» attached to the next word. The same normalization is applied to references and hypotheses.
95 % confidence intervals come from a cluster bootstrap over recordings.
Against Vosk: this model makes fewer character errors (CER 12.0 vs 15.7) but more word errors. Its largest error category (about 31 %) is near-miss spelling (see Limitations).

Examples from the test set:

Table with columns: Reference, This model
Reference	This model
كيفاش أحنا on va changer le process	كيفاش أحنا on va changer le process
معانا محرز بالتليفون مرحبا	معانا محرز بالتليفون مرحبا
يقولوا كون اليابان بلاد غنية	يقولوا كونوا اليابان بلاد غنية

How to use

python

import librosafrom transformers import WhisperProcessor, WhisperForConditionalGeneration repo = "TheBiggyBang/whisper-small-tunisian-derja"processor = WhisperProcessor.from_pretrained(repo)model = WhisperForConditionalGeneration.from_pretrained(repo) audio, _ = librosa.load("audio.wav", sr=16000)          # mono, 16 kHz, up to 30 sinputs = processor(audio, sampling_rate=16000, return_tensors="pt")ids = model.generate(inputs.input_features, language="arabic", task="transcribe", num_beams=3)print(processor.batch_decode(ids, skip_special_tokens=True)[0])

For recordings longer than 30 s, use the pipeline with chunking:

python

from transformers import pipeline asr = pipeline("automatic-speech-recognition", model="TheBiggyBang/whisper-small-tunisian-derja", chunk_length_s=30)print(asr("long_audio.wav", generate_kwargs={"language": "arabic", "task": "transcribe"})["text"])

For speed, CPU deployment or live microphone transcription, use the CTranslate2 version.

Output style: Derja in Arabic script. French/English words appear in Latin script (or Arabic script, as the annotators sometimes wrote them). Punctuation is kept and diacritics are removed.

Training data

Trained on the LinTO Tunisian datasets by LINAGORA, both licensed CC-BY-4.0. The default config of each was streamed (all sub-corpora):

Table with columns: Dataset, Share of training samples
Dataset	Share of training samples
linagora/linto-dataset-audio-ar-tn (`train`)	40 %
linagora/linto-dataset-audio-ar-tn-augmented (`train`, voice-converted and denoised copies of the train split)	60 %

These include TunSwitch CS/TO, MASC, AmenyKH, ApprendreLeTunisien, OneStory, Tunisian_dataset_STT-TTS15s, Wav2Vec-tunisian-Darja and several YouTube collections. See the dataset cards for each source.

Leakage controls:

Every evaluation sentence with ≥ 3 words (679 sentences) is block-listed. Training recordings that contain one are dropped (97 recordings).
Dev and test are split by recording, not by segment.

Training procedure

Table with columns: Setting, Value
Setting	Value
Hardware	2× NVIDIA T4 (Kaggle), 10.3 h
Steps	4,071 × effective batch 32 (≈ 130 k segments); best checkpoint at step 3,416, chosen by dev normalized WER
Optimizer	AdamW, lr 1e-5, cosine schedule, 300 warm-up steps, weight decay 0.01, grad-clip 1.0, fp16
Augmentation	SpecAugment (time masks), random gain ±6 dB, coloured noise at SNR 15–30 dB
Segments	0.6–29 s, ≤ 220 tokens, ≤ 4 segments per recording
Prompt	`language=arabic`, `task=transcribe`, no timestamps

Evaluation protocol

Pool: all six original test splits (AmenyKH, ApprendreLeTunisien, OneStory, TunSwitchCS, TunSwitchTO, Youtube_TNScrapped_V1).
Split: by recording into dev (323 clips, used for checkpoint selection) and test (482 clips, used once).
Test composition: TunSwitchCS 185 · TunSwitchTO 171 · ApprendreLeTunisien 69 · AmenyKH 30 · Youtube_TNScrapped_V1 15 · OneStory 12.
Metrics are corpus-level (total edits / total reference words).

Limitations

Speaker overlap is possible. The datasets have no speaker IDs, so the test set is recording-disjoint but not speaker-disjoint. Real-world accuracy on new speakers may be lower.
Spelling. Derja has no standard orthography. The largest error category is near-miss spelling (e.g. لي/الي, موش/مش, والا/ولا), so report CER alongside WER.
Code-switching. French/English words may come out in a different script than a human annotator would use.
Small test set (36.6 min). Per-corpus numbers are not reported because some sub-corpora have only 12–15 test clips.
Inherited Whisper risks. Hallucinations and repetition are possible on noisy or unclear audio. Do not use it on recordings made without consent or for high-stakes decisions.

Attribution

Fine-tuned from openai/whisper-small (Apache-2.0). Training data: the LinTO DataSet Audio for Arabic Tunisian and its augmented version by LINAGORA, licensed under CC-BY-4.0. These include TunSwitch (Abdallah et al., 2023), MASC (Al-Fetyani et al., 2021) and other community and web sources listed on the dataset cards. The data was segmented, filtered and used for training; no audio or transcripts are redistributed here.

Citation

If you use this model, please cite the works it builds on:

bibtex

@misc{linagora2024Linto-tn,  title = {LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect},  author = {Hedi Naouara and Jérôme Louradour and Jean-Pierre Lorré},  year = {2025},  month = {March},  eprint={2504.02604},  archivePrefix={arXiv},  primaryClass={cs.CL},  note={Good Data Workshop, AAAI 2025},  url={arxiv.org/abs/2504.02604},} @misc{abdallah2023leveraging,      title={Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition},      author={Ahmed Amine Ben Abdallah and Ata Kabboudi and Amir Kanoun and Salah Zaiem},      year={2023},      eprint={2309.11327},      archivePrefix={arXiv},      primaryClass={eess.AS}} @data{e1qb-jv46-21,doi = {10.21227/e1qb-jv46},url = {https://dx.doi.org/10.21227/e1qb-jv46},author = {Al-Fetyani, Mohammad and Al-Barham, Muhammad and Abandah, Gheith and Alsharkawi, Adham and Dawas, Maha},publisher = {IEEE Dataport},title = {MASC: Massive Arabic Speech Corpus},year = {2021} } @misc{radford2022whisper,  doi = {10.48550/ARXIV.2212.04356},  url = {https://arxiv.org/abs/2212.04356},  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},  title = {Robust Speech Recognition via Large-Scale Weak Supervision},  publisher = {arXiv},  year = {2022},  copyright = {arXiv.org perpetual, non-exclusive license}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

TheBiggyBang

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Results

Held-out test set: 482 clips (36.6 min) drawn from all LinTO test splits. No recording in it was seen during training or checkpoint selection.

Table with columns: Model / decoder, WER, normalized WER (95 % CI), CER
Model / decoder	WER	normalized WER (95 % CI)	CER
whisper-small (no fine-tuning), CT2 beam 3	79.1	76.3 (73.8–79.0)	44.5
this model, Transformers fp16 greedy	32.8	31.9 (29.9–34.2)	13.4
this model, CTranslate2 int8 beam 3	30.4	29.5 (27.5–31.6)	12.0
LinTO Kaldi/Vosk (with 4-gram LM, scored on 400 of the 482 clips)	25.4	25.0 (22.0–28.1)	15.7

Normalized WER removes Derja spelling variants that are not recognition errors: أ/إ/آ→ا, ى→ي, ة→ه, Arabic-Indic digits, punctuation, lowercase Latin, and standalone «و» attached to the next word. The same normalization is applied to references and hypotheses.
95 % confidence intervals come from a cluster bootstrap over recordings.
Against Vosk: this model makes fewer character errors (CER 12.0 vs 15.7) but more word errors. Its largest error category (about 31 %) is near-miss spelling (see Limitations).

Examples from the test set:

Table with columns: Reference, This model
Reference	This model
كيفاش أحنا on va changer le process	كيفاش أحنا on va changer le process
معانا محرز بالتليفون مرحبا	معانا محرز بالتليفون مرحبا
يقولوا كون اليابان بلاد غنية	يقولوا كونوا اليابان بلاد غنية

How to use

python

import librosafrom transformers import WhisperProcessor, WhisperForConditionalGeneration repo = "TheBiggyBang/whisper-small-tunisian-derja"processor = WhisperProcessor.from_pretrained(repo)model = WhisperForConditionalGeneration.from_pretrained(repo) audio, _ = librosa.load("audio.wav", sr=16000)          # mono, 16 kHz, up to 30 sinputs = processor(audio, sampling_rate=16000, return_tensors="pt")ids = model.generate(inputs.input_features, language="arabic", task="transcribe", num_beams=3)print(processor.batch_decode(ids, skip_special_tokens=True)[0])

import librosafrom transformers import WhisperProcessor, WhisperForConditionalGeneration repo = "TheBiggyBang/whisper-small-tunisian-derja"processor = WhisperProcessor.from_pretrained(repo)model = WhisperForConditionalGeneration.from_pretrained(repo) audio, _ = librosa.load("audio.wav", sr=16000)          # mono, 16 kHz, up to 30 sinputs = processor(audio, sampling_rate=16000, return_tensors="pt")ids = model.generate(inputs.input_features, language="arabic", task="transcribe", num_beams=3)print(processor.batch_decode(ids, skip_special_tokens=True)[0])

For recordings longer than 30 s, use the pipeline with chunking:

python

from transformers import pipeline asr = pipeline("automatic-speech-recognition", model="TheBiggyBang/whisper-small-tunisian-derja", chunk_length_s=30)print(asr("long_audio.wav", generate_kwargs={"language": "arabic", "task": "transcribe"})["text"])

from transformers import pipeline asr = pipeline("automatic-speech-recognition", model="TheBiggyBang/whisper-small-tunisian-derja", chunk_length_s=30)print(asr("long_audio.wav", generate_kwargs={"language": "arabic", "task": "transcribe"})["text"])

For speed, CPU deployment or live microphone transcription, use the CTranslate2 version.

Output style: Derja in Arabic script. French/English words appear in Latin script (or Arabic script, as the annotators sometimes wrote them). Punctuation is kept and diacritics are removed.

Training data

Trained on the LinTO Tunisian datasets by LINAGORA, both licensed CC-BY-4.0. The default config of each was streamed (all sub-corpora):

Table with columns: Dataset, Share of training samples
Dataset	Share of training samples
linagora/linto-dataset-audio-ar-tn (`train`)	40 %
linagora/linto-dataset-audio-ar-tn-augmented (`train`, voice-converted and denoised copies of the train split)	60 %

Leakage controls:

Every evaluation sentence with ≥ 3 words (679 sentences) is block-listed. Training recordings that contain one are dropped (97 recordings).
Dev and test are split by recording, not by segment.

Training procedure

Table with columns: Setting, Value
Setting	Value
Hardware	2× NVIDIA T4 (Kaggle), 10.3 h
Steps	4,071 × effective batch 32 (≈ 130 k segments); best checkpoint at step 3,416, chosen by dev normalized WER
Optimizer	AdamW, lr 1e-5, cosine schedule, 300 warm-up steps, weight decay 0.01, grad-clip 1.0, fp16
Augmentation	SpecAugment (time masks), random gain ±6 dB, coloured noise at SNR 15–30 dB
Segments	0.6–29 s, ≤ 220 tokens, ≤ 4 segments per recording
Prompt	`language=arabic`, `task=transcribe`, no timestamps

Evaluation protocol

Pool: all six original test splits (AmenyKH, ApprendreLeTunisien, OneStory, TunSwitchCS, TunSwitchTO, Youtube_TNScrapped_V1).
Split: by recording into dev (323 clips, used for checkpoint selection) and test (482 clips, used once).
Test composition: TunSwitchCS 185 · TunSwitchTO 171 · ApprendreLeTunisien 69 · AmenyKH 30 · Youtube_TNScrapped_V1 15 · OneStory 12.
Metrics are corpus-level (total edits / total reference words).

Limitations

Speaker overlap is possible. The datasets have no speaker IDs, so the test set is recording-disjoint but not speaker-disjoint. Real-world accuracy on new speakers may be lower.
Spelling. Derja has no standard orthography. The largest error category is near-miss spelling (e.g. لي/الي, موش/مش, والا/ولا), so report CER alongside WER.
Code-switching. French/English words may come out in a different script than a human annotator would use.
Small test set (36.6 min). Per-corpus numbers are not reported because some sub-corpora have only 12–15 test clips.
Inherited Whisper risks. Hallucinations and repetition are possible on noisy or unclear audio. Do not use it on recordings made without consent or for high-stakes decisions.

Attribution

Citation

If you use this model, please cite the works it builds on:

bibtex

@misc{linagora2024Linto-tn,  title = {LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect},  author = {Hedi Naouara and Jérôme Louradour and Jean-Pierre Lorré},  year = {2025},  month = {March},  eprint={2504.02604},  archivePrefix={arXiv},  primaryClass={cs.CL},  note={Good Data Workshop, AAAI 2025},  url={arxiv.org/abs/2504.02604},} @misc{abdallah2023leveraging,      title={Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition},      author={Ahmed Amine Ben Abdallah and Ata Kabboudi and Amir Kanoun and Salah Zaiem},      year={2023},      eprint={2309.11327},      archivePrefix={arXiv},      primaryClass={eess.AS}} @data{e1qb-jv46-21,doi = {10.21227/e1qb-jv46},url = {https://dx.doi.org/10.21227/e1qb-jv46},author = {Al-Fetyani, Mohammad and Al-Barham, Muhammad and Abandah, Gheith and Alsharkawi, Adham and Dawas, Maha},publisher = {IEEE Dataport},title = {MASC: Massive Arabic Speech Corpus},year = {2021} } @misc{radford2022whisper,  doi = {10.48550/ARXIV.2212.04356},  url = {https://arxiv.org/abs/2212.04356},  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},  title = {Robust Speech Recognition via Large-Scale Weak Supervision},  publisher = {arXiv},  year = {2022},  copyright = {arXiv.org perpetual, non-exclusive license}}

@misc{linagora2024Linto-tn,  title = {LinTO Audio and Textual Datasets to Train and Evaluate Automatic Speech Recognition in Tunisian Arabic Dialect},  author = {Hedi Naouara and Jérôme Louradour and Jean-Pierre Lorré},  year = {2025},  month = {March},  eprint={2504.02604},  archivePrefix={arXiv},  primaryClass={cs.CL},  note={Good Data Workshop, AAAI 2025},  url={arxiv.org/abs/2504.02604},} @misc{abdallah2023leveraging,      title={Leveraging Data Collection and Unsupervised Learning for Code-switched Tunisian Arabic Automatic Speech Recognition},      author={Ahmed Amine Ben Abdallah and Ata Kabboudi and Amir Kanoun and Salah Zaiem},      year={2023},      eprint={2309.11327},      archivePrefix={arXiv},      primaryClass={eess.AS}} @data{e1qb-jv46-21,doi = {10.21227/e1qb-jv46},url = {https://dx.doi.org/10.21227/e1qb-jv46},author = {Al-Fetyani, Mohammad and Al-Barham, Muhammad and Abandah, Gheith and Alsharkawi, Adham and Dawas, Maha},publisher = {IEEE Dataport},title = {MASC: Massive Arabic Speech Corpus},year = {2021} } @misc{radford2022whisper,  doi = {10.48550/ARXIV.2212.04356},  url = {https://arxiv.org/abs/2212.04356},  author = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},  title = {Robust Speech Recognition via Large-Scale Weak Supervision},  publisher = {arXiv},  year = {2022},  copyright = {arXiv.org perpetual, non-exclusive license}}