Congo-digital-service

whisper-small-lingala-qlora-merged

README

License: apache-2.0

Summary

Table

Task	Speech-to-text transcription of spoken Lingala
Base model	`openai/whisper-small` (Apache 2.0)
Method	QLoRA (4-bit NF4 base model + LoRA adapters, rank 32), then adapters merged into the base weights
Training data	Train split of `Congo-digital-service/audios-lingala-annotatees-v2`
Reference result	WER 44.17 %, CER 18.95 % on the leak-free v3.2 test set (1,797 segments, harmonised reading C)
Earlier result	WER 46.4 %, CER 19.4 % on the v2 test set (1,882 segments)
Status	Work-in-progress model; the inference endpoint is not yet validated (see Validation status)
Licence	Apache 2.0 (model weights); see Licence for the training data
Developed by	Congo Digital Services (CDS SARL), congo-digital.com

Uses

Direct use: transcription of Lingala speech, either with transformers or through a compatible inference server exposing an OpenAI-style transcription endpoint (see Deployment).

Out of scope:

production deployment without review, since the model and its endpoint are not yet validated;
high-stakes transcription (legal, medical, safety-critical audio) without human review;
other languages;
speaker identification.

How to use

python

import librosaimport torchfrom transformers import WhisperForConditionalGeneration, WhisperProcessor model_id = "Congo-digital-service/whisper-small-lingala-qlora-merged"device = "cuda" if torch.cuda.is_available() else "cpu" processor = WhisperProcessor.from_pretrained(model_id)model = WhisperForConditionalGeneration.from_pretrained(model_id).to(device).eval() audio, _ = librosa.load("audio.wav", sr=16000)  # Whisper expects 16 kHz monofeatures = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to(device)with torch.no_grad():    ids = model.generate(input_features=features, language="ln", task="transcribe")print(processor.batch_decode(ids, skip_special_tokens=True)[0])

Always resample to 16 kHz, and pass language="ln" and task="transcribe". Without the language token, Whisper guesses the language and may transcribe or translate into another one.

Training data

Dataset: Congo-digital-service/audios-lingala-annotatees-v2, train split: annotated Lingala speech, segmented and transcribed by MALOBA annotators.
Spelling: transcriptions follow the annotators' spelling. They are not orthographically harmonised: the same word can appear in several spellings, and the Lingala letters ɔ and ɛ are rarely written.
Known data issue: 17 test segments of v2 have audio identical to segments in the training split. The leak-free v3.2 test set removes them (see Evaluation).

Training procedure

QLoRA on openai/whisper-small: 4-bit NF4 quantisation of the base model; LoRA rank 32, alpha 64, dropout 0.05, on q_proj, k_proj, v_proj, out_proj, fc1, fc2.
2,500 steps, with evaluation every 250 steps. The checkpoint with the best validation WER was retained, then the adapters were merged and the merged model exported.
Framework: PEFT 0.20.0, transformers.

Training-time validation curve (original training notebook, its own protocol: augmented audio, no text normalisation). These values are not the reference scores, which are given under Evaluation.

Table with columns: Step, Training loss, Validation loss, WER, CER
Step	Training loss	Validation loss	WER	CER
250	1.2962	1.3415	64.45 %	26.53 %
500	1.2138	1.1822	60.29 %	24.74 %
750	1.1461	1.0893	57.63 %	23.60 %
1,000

Known recipe defects, identified after training and documented for a future retraining: the language token was missing from the decoder prefix; the start-of-transcript token was duplicated in the labels; the audio augmentation (±4 semitones) was frozen, so the model never heard unmodified audio; only 64 % of the available segments were used.

Evaluation

Reference: leak-free v3.2 test set (29 September 2026)

Test set: audios-lingala-annotatees-v3.2 @ fed93c0e, split test, 1,797 segments from 19 source recordings, 4.83 h, all marked eval_propre: no segment shares its audio with the v2 training data (two independent checks).
Three readings of the same outputs: A, original reference against raw output; B, harmonised reference against raw output; C (reference reading), harmonised reference against harmonised output (outils/harmonisation_maloba.py).
Intervals: bootstrap by source recording, 1,000 resamples, seed 12345.
Weights: the adapters at revision 501d57ff, which are the weights merged into this model (adapter_model.safetensors is identical at revisions 501d57ff and 0e0b6fc2).

Table with columns: Model, WER, reading A, WER, reading C, 95 % CI, CER, reading C, 95 % CI
Model	WER, reading A	WER, reading C	95 % CI	CER, reading C	95 % CI
MMS 1B + MALOBA Lingala adapter	47.35 %	44.12 %	41.32–47.04	16.26 %	14.91–17.59
This model	47.45 %	44.17 %	40.71–47.78	18.95 %	16.90–20.77
, stock Lingala adapter

Spelling cost: between readings A and C the outputs are identical; 3.28 WER points (6.9 % of the reading-A WER) come from spelling convention, not transcription.
Against the fine-tuned MMS adapter: −0.09 point, 95 % CI [−2.12, +1.87]: no established difference at the word level. At the character level, MMS makes fewer errors (16.26 % against 18.95 %).

Earlier measurement: v2 test set (16 September 2026)

Run 2026-09-16T11-18-09Z, 1,882 segments from 20 recordings, micro aggregation; normalised form: Unicode NFC, ɔ/ɛ/ε neutralised.

Table with columns: Model, WER raw, WER norm., CER raw, CER norm.
Model	WER raw	WER norm.	CER raw	CER norm.
`openai/whisper-small` (base)	162.3 %	196.0 %	106.2 %	102.4 %
`facebook/mms-1b-all`, Lingala adapter	60.4 %	60.1 %	23.3 %	23.2 %
This model	47.3 %

This set contains 17 segments seen in training, and its references are not harmonised: the v3.2 figures above replace it as the reference. The two are not directly comparable (different segments, recordings and spelling).

External reference: FLEURS Lingala

google/fleurs, ln_cd, test split, 478 read sentences: WER 39.39 %, CER 15.36 % (normalised). FLEURS is read, studio-quality speech with a different spelling convention (no ɔ/ɛ, tone marks); it validates the protocol but does not predict performance on field recordings.

Other references

Human reference floor: two annotators transcribing the same audio disagree by 12.1 % CER.
Published result: Nahabwe et al. (2025) report 19.85 % WER for fine-tuned XLS-R on about 50 h of Lingala, on a different test set: a reference point, not a comparison.

Human evaluation (3 validators, 25 outputs on the v2 test set): 16 % judged correct by majority, 100 % recoverable, 0 % incorrect. Inter-rater agreement is too low for a published rate (Krippendorff's α = 0.258, below the 0.60 threshold set before the campaign). A new campaign on the v3.2 test set is under way.

Superseded figure: the 53.38 % WER / 22.12 % CER previously published on this card are training-time validation values. They are withdrawn as reference scores.

Deployment

Table with columns: Component, Detail
Component	Detail
Inference server	OpenAI-compatible transcription server (for example vLLM with audio support)
Endpoint	`POST /v1/audio/transcriptions`, `multipart/form-data`
Inputs	audio file, model identifier, `language`, optional `prompt`, `response_format`, `temperature`
Output	`{"text": …}`, or detailed JSON
MALOBA gateway

Validation status

The merged model is available on Hugging Face. The end-to-end endpoint (gateway → inference server → model) is in place on the gateway side and remains to be validated, with latency measurement, once the inference server is deployed.

Limitations

High error rate: 44 % WER on field speech. Transcriptions are drafts that need human review. The project's internal target of 15 % WER predates the measurement of the 12.1 % human floor, and is a CDS target, not a requirement of the terms of reference.
Spelling variation in the training transcriptions: about 7 % of the WER on the v3.2 test set comes from the spelling convention alone.
Small effective sample: 19 test recordings; a difference of less than 3–4 WER points cannot be established on this set.
Training recipe: see the known defects under Training procedure. A retraining on the harmonised v3.2 corpus with a corrected recipe is recommended.
Domain: radio and field speech. Accents, dialects and recording conditions absent from the training data are not evaluated.

Environmental impact

Training ran on a single GPU (Google Colab) for 2,500 steps. The hardware type and energy use were not recorded; see the Machine Learning Impact calculator (Lacoste et al., 2019).

Licence

Model weights: Apache 2.0, the licence of openai/whisper-small.
Training data: the MALOBA speech corpus v2 is being aligned on the Nwulite Obodo Open Data License (NOODL-1.0), already applied to v3.2. CDS asks users from high-income countries and commercial entities to publicly credit the MALOBA Project (UNDP Republic of Congo — language digitalisation initiative) in any derived publication, product, model or output. Contact: contact@congo-digital.com.

Citation

bibtex

@misc{cds2026whisperlingala,  title        = {whisper-small-lingala-qlora},  author       = {{Congo Digital Services}},  year         = {2026},  howpublished = {\url{https://huggingface.co/Congo-digital-service/whisper-small-lingala-qlora-merged}},  note         = {MALOBA project, UNDP Republic of Congo}} @inproceedings{radford2023whisper,  title     = {Robust Speech Recognition via Large-Scale Weak Supervision},  author    = {Radford, Alec and Kim, Jong Wook and Xu, Tao and Brockman, Greg and McLeavey, Christine and Sutskever, Ilya},  booktitle = {Proceedings of the 40th International Conference on Machine Learning},  pages     = {28492--28518},  year      = {2023}}

Contact

Organisation: Congo Digital Services (CDS SARL)
Website: congo-digital.com
Email: contact@congo-digital.com

Card updated on 30 September 2026.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Congo-digital-service

Model Tree

Base

openai/whisper-small

Fine-tuned

this model

Input Modalities

Audio

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer