Congo-digital-service

qwen-vl-lingala-qlora-vf

README

License: apache-2.0

Summary

Table

Task	Transcription of Lingala text in images (OCR)
Base model	`Qwen/Qwen2-VL-2B-Instruct` (Apache 2.0)
Method	QLoRA (LoRA rank 32), four tokens added (Ɔ Ɛ ɔ ɛ), then adapters merged
Training data	417 image–text examples (278 lines, ɔ/ɛ lines oversampled) from `dataset-qwen-vl-lingala-qlora-vf`
Test result	CER 9.4 %, i.e. 90.6 % of characters correct (normalised, micro, 50 images); raw CER 9.8 %; 34 of the 43 ɛ/ɔ of the references produced
Human evaluation	32 % of outputs judged correct, 92 % recoverable (3 validators, 25 images, agreement α = 0.665), on outputs of the previous revision
Licence	Apache 2.0, the licence of the base model
Developed by	Congo Digital Services (CDS SARL), congo-digital.com

Revision note: output head fix (October 2026)

At revision 42ac8f8b079ea5babad0efb5b5417b5e546499da, model.safetensors stored a fine-tuned lm_head.weight that differs from model.embed_tokens.weight, while config.json declared tied word embeddings. Libraries resolved this differently: transformers 4.57 tied the head to the embeddings and ignored the stored lm_head, so the model never emitted ɛ/ɔ; transformers 5.x and vLLM 0.30 loaded the stored lm_head. Details in discussion #1.

This revision ships a hybrid output head:

lm_head.weight equals model.embed_tokens.weight, except rows 151657 to 151660 (ɔ ɛ Ɔ Ɛ, the four tokens added during fine-tuning), copied from the fine-tuned lm_head. No other weight changes.
tie_word_embeddings: false, at top level and in text_config, so that every library loads this same head.

Official measurement of 22/10/2026 (MALOBA evaluation chain, run 2026-10-22T09-35-40Z): test split of dataset-qwen-vl-lingala-qlora-vf (50 images), training prompt, greedy decoding, max_new_tokens=256, same package versions as the 16/09 measurement; both heads in the same run.

Table with columns: Tied head (b54def0b), Hybrid head (this revision)
	Tied head (`b54def0b`)	Hybrid head (this revision)
CER, normalised (micro)	11.5 %	9.4 %
CER, raw (micro)	69.6 %	9.8 %
Median CER, normalised	7.4 %	1.7 %
Exact transcriptions, normalised	28 %	44 %
Exact transcriptions, raw	24 %	38 %
WER, normalised (micro)	23.3 %	20.9 %
ɛ/ɔ produced (43 in the references)	0	34
Looping outputs	2	1

The tied-head result reproduces the 16/09 measurement (11.5 % vs 11.3 %; 45 of 50 outputs identical, the others differ because of the GPU and torch version); the 50 test references are unchanged. Normalisation maps ɛ to e and ɔ to o, so the normalised CER understates the gain on these letters. The hybrid head was selected on this same test set: its advantage is slightly optimistic and should be confirmed on held-out documents.

Uses

In scope: transcription of Lingala text lines photographed or scanned from documents similar to the training data (printed and handwritten material from the SIL Congo collection), with human review of the output.

Out of scope:

unreviewed transcription for official, legal or medical records;
full pages or complex layouts: the model was trained on single lines;
other languages and scripts;
high-volume production without a limit on output length (see Limitations).

How to use

python

import torchfrom PIL import Imagefrom transformers import Qwen2VLForConditionalGeneration, AutoProcessor model_id = "Congo-digital-service/qwen-vl-lingala-qlora-vf"model = Qwen2VLForConditionalGeneration.from_pretrained(    model_id, torch_dtype=torch.bfloat16, device_map="auto").eval()processor = AutoProcessor.from_pretrained(model_id) image = Image.open("line.png").convert("RGB")messages = [{"role": "user", "content": [    {"type": "image"},    {"type": "text", "text": "Transcris le texte visible sur cette image, en lingala."},]}]prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device) with torch.no_grad():    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)new_tokens = out[:, inputs["input_ids"].shape[1]:]print(processor.batch_decode(new_tokens, skip_special_tokens=True)[0])

Always send this exact prompt, the training prompt, with greedy decoding: they are the settings of every score below. With a free-form instruction (for example « ocrise »), the model leaves its fine-tuning: it answers in English, guesses a language, and loses ɛ/ɔ. Keep max_new_tokens bounded: a few outputs loop (see Limitations).

Training data

Source: line-level regions annotated in Label Studio from the SIL Congo document collection (printed and handwritten Lingala).
Sample: 164 lines containing ɔ or ɛ, plus 164 ordinary lines, i.e. 328 lines, drawn before the annotation campaign was complete.
Split: 85/15, before any oversampling: 278 training lines, 50 test lines (48 distinct texts in the test set).
Oversampling: the ɔ/ɛ lines of the training set were duplicated (×2), giving 417 training examples. The test set is unchanged.
Wider corpus: the annotation campaign has since produced 1,323 unique human-validated transcriptions, 1,282 of them harmonised to the MALOBA spelling convention. They were not used for this model.

Training procedure

Table with columns: Setting, Value
Setting	Value
Method	QLoRA, PagedAdamW 8-bit
LoRA	rank 32, alpha 64, dropout 0.1
LoRA targets	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, and the vision merger (`merger.mlp.0`, `merger.mlp.2`)
Fully trained modules	`embed_tokens`, `lm_head` (needed for the four added tokens)
Added tokens	`Ɔ` `Ɛ` `ɔ` `ɛ`
Epochs	5
Learning rate	1 × 10⁻⁴, linear schedule, 10 % warm-up
Batch	1 × 8 gradient accumulation
Environment	Google Colab, August 2026

Per-epoch scores computed by the training notebook on the 50 test images (not the reference scores):

Table with columns: Epoch, Training loss, CER, WER, Exact match
Epoch	Training loss	CER	WER	Exact match
1	1.6905	17.04 %	30.86 %	34.0 %
2	0.2428	28.06 %	37.94 %	48.0 %
3	0.0916	18.96 %	32.04 %	40.0 %
4	0.0533	26.92 %	43.00 %	40.0 %
5	0.0360	19.44 %	31.03 %	44.0 %

Training loss collapses while the test CER does not improve: overfitting from epoch 2. The epoch-1 checkpoint was retained. This choice was made on the test set, as no separate validation set was kept; the published score is therefore slightly optimistic.

Evaluation

Protocol (MALOBA evaluation chain): 50 test images, same prompt and greedy decoding for every model, micro aggregation (errors summed over the whole set). Normalised form: Unicode NFC, ɔ/ɛ/ε neutralised on both sides; the raw form is reported alongside. Base model: run 2026-09-16T12-57-27Z. Previous revision and this revision: run 2026-10-22T09-35-40Z.

Table with columns: Metric, Base model, Previous revision (tied head), This revision (hybrid head)
Metric	Base model	Previous revision (tied head)	This revision (hybrid head)
CER, normalised (micro)	40.0 %	11.5 %	9.4 % (90.6 % of characters correct)
CER, raw (micro)	44.6 %	69.6 %	9.8 %
Median CER, normalised	—	7.4 %	1.7 %
WER, normalised (micro)	—	23.3 %	20.9 %
Exact match, normalised	24.0 %	28 %	44 %
ɛ/ɔ produced (43 in the references)	—	0	34
Looping outputs	—	2	1

Raw and normalised CER. With the tied head, a few outputs loop up to 2,000 characters for a 24-character reference, and these loops dominated the raw micro CER (69.3 % on 16/09). With the hybrid head, one output loops and the raw and normalised CER are close.

Human evaluation (3 Lingala-speaking validators, 25 test images, majority verdict; outputs of the previous revision, with the tied head):

Table with columns: Verdict, Share, 95 % CI
Verdict	Share	95 % CI
Correct	32 %	17–52 %
Correct or partially correct (recoverable)	92 %	75–98 %
Incorrect	8 %	—

Inter-rater agreement is α = 0.665, above the 0.60 threshold set before the campaign: this result is usable. The error validators cite most is "wrong ɔ/ɛ character" (43 mentions out of 50), ahead of invented text (21) and ordinary character errors (19). The automatic metric neutralises these letters; the validators do not.

Limitations

Looping outputs: 1 in 50 outputs with this revision (2 to 3 with the tied head) repeats until the length limit. Always bound max_new_tokens; a repetition check at inference removes the problem without retraining.
ɔ and ɛ: this revision produces 34 of the 43 expected in the test references. The human evaluation was run on outputs of the tied head, which could not emit these letters at all, and has not been repeated.
Invalid UTF-8 bytes appear in place of "ɔ́" (open o with an accent) in about 1 output in 25: an interaction between the added tokens and combining accents.
Small test set: 50 images. Near 95 % accuracy, the uncertainty is about ±6 points; this set cannot prove or rule out a 95 % target.
Checkpoint and output head chosen on the test set (see Training procedure and the revision note).
Line-level only: trained on single text lines, not full pages.

Deployment

In the MALOBA inference gateway (LiteLLM, OpenAI-compatible), this model is served under the alias maloba-ocr, route POST /v1/chat/completions, with the image sent as a base64 image_url and the prompt above. The context must allow at least 4,096 tokens.

Licence

Model weights: Apache 2.0, the licence of Qwen/Qwen2-VL-2B-Instruct.
Licence text and notice: LICENSE (Apache 2.0) and NOTICE (base model and changes made).
Training data: the MALOBA OCR dataset is published under the Nwulite Obodo Open Data License (NOODL-1.0). CDS asks users from high-income countries and commercial entities to publicly credit the MALOBA project (UNDP Republic of Congo — language digitalisation initiative) in any publication, product, model or output derived from this model. Contact: contact@congo-digital.com.

Citation

bibtex

@misc{cds2026qwenvllingala,  title        = {qwen-vl-lingala-qlora-vf: Lingala text recognition with Qwen2-VL},  author       = {{Congo Digital Services}},  year         = {2026},  howpublished = {\url{https://huggingface.co/Congo-digital-service/qwen-vl-lingala-qlora-vf}},  note         = {MALOBA project, UNDP Republic of Congo}}

Creators

Congo Digital Services (CDS SARL): https://www.congo-digital.com/
In collaboration with the MALOBA community (https://maloba.congo-digital.com/), SIL Congo (document collection), the Ministry of Posts, Telecommunications and the Digital Economy of the Republic of Congo and UNDP Congo.

Created in August 2026. Card updated on 22 October 2026 (output head fix, discussion #1).

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Congo-digital-service

Model Tree

Base

Qwen/Qwen2-VL-2B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality