Congo-digital-service

qwen-vl-lingala-qlora-vf

README

License: apache-2.0

Summary

Table

Task	Transcription of Lingala text in images (OCR)
Base model	`Qwen/Qwen2-VL-2B-Instruct` (Apache 2.0)
Method	QLoRA (LoRA rank 32), four tokens added (Ɔ Ɛ ɔ ɛ), then adapters merged
Training data	417 image–text examples (278 lines, ɔ/ɛ lines oversampled) from `dataset-qwen-vl-lingala-qlora-vf`
Test result	CER 11.3 %, i.e. 88.7 % of characters correct (normalised, micro, 50 images)
Human evaluation	32 % of outputs judged correct, 92 % recoverable (3 validators, 25 images, agreement α = 0.665)
Licence	Apache 2.0, the licence of the base model
Developed by	Congo Digital Services (CDS SARL), congo-digital.com

Uses

In scope: transcription of Lingala text lines photographed or scanned from documents similar to the training data (printed and handwritten material from the SIL Congo collection), with human review of the output.

Out of scope:

unreviewed transcription for official, legal or medical records;
full pages or complex layouts: the model was trained on single lines;
other languages and scripts;
high-volume production without a limit on output length (see Limitations).

How to use

python

import torchfrom PIL import Imagefrom transformers import Qwen2VLForConditionalGeneration, AutoProcessor model_id = "Congo-digital-service/qwen-vl-lingala-qlora-vf"model = Qwen2VLForConditionalGeneration.from_pretrained(    model_id, torch_dtype=torch.bfloat16, device_map="auto").eval()processor = AutoProcessor.from_pretrained(model_id) image = Image.open("line.png").convert("RGB")messages = [{"role": "user", "content": [    {"type": "image"},    {"type": "text", "text": "Transcris le texte de cette image."},]}]prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device) with torch.no_grad():    out = model.generate(**inputs, max_new_tokens=256, do_sample=False)new_tokens = out[:, inputs["input_ids"].shape[1]:]print(processor.batch_decode(new_tokens, skip_special_tokens=True)[0])

Use this prompt and greedy decoding: they are the settings of every score below. Keep max_new_tokens bounded: a few outputs loop (see Limitations).

Training data

Source: line-level regions annotated in Label Studio from the SIL Congo document collection (printed and handwritten Lingala).
Sample: 164 lines containing ɔ or ɛ, plus 164 ordinary lines, i.e. 328 lines, drawn before the annotation campaign was complete.
Split: 85/15, before any oversampling: 278 training lines, 50 test lines (48 distinct texts in the test set).
Oversampling: the ɔ/ɛ lines of the training set were duplicated (×2), giving 417 training examples. The test set is unchanged.
Wider corpus: the annotation campaign has since produced 1,323 unique human-validated transcriptions, 1,282 of them harmonised to the MALOBA spelling convention. They were not used for this model.

Training procedure

Table with columns: Setting, Value
Setting	Value
Method	QLoRA, PagedAdamW 8-bit
LoRA	rank 32, alpha 64, dropout 0.1
LoRA targets	`q_proj`, `k_proj`, `v_proj`, `o_proj`, `gate_proj`, `up_proj`, `down_proj`, and the vision merger (`merger.mlp.0`, `merger.mlp.2`)
Fully trained modules

Per-epoch scores computed by the training notebook on the 50 test images (not the reference scores):

Table with columns: Epoch, Training loss, CER, WER, Exact match
Epoch	Training loss	CER	WER	Exact match
1	1.6905	17.04 %	30.86 %	34.0 %
2	0.2428	28.06 %	37.94 %	48.0 %
3	0.0916	18.96 %	32.04 %	40.0 %

Training loss collapses while the test CER does not improve: overfitting from epoch 2. The epoch-1 checkpoint was retained. This choice was made on the test set, as no separate validation set was kept; the published score is therefore slightly optimistic.

Evaluation

Protocol (MALOBA evaluation chain, run 2026-09-16T12-57-27Z): 50 test images, same prompt and greedy decoding for the base and fine-tuned models, micro aggregation (errors summed over the whole set). Normalised form: Unicode NFC, ɔ/ɛ/ε neutralised on both sides; the raw form is reported alongside.

Table with columns: Metric, Base model, This model
Metric	Base model	This model
CER, normalised (micro)	40.0 %	11.3 % (88.7 % of characters correct)
CER, raw (micro)	44.6 %	69.3 %
WER, normalised (micro)	—	19.9 %
Exact match, normalised	24.0 %	28.0 % (95 % CI 17.5–41.7)

Why the raw CER is worse than the base. Three outputs out of fifty loop, producing up to 2,000 characters for a 24-character reference. The micro raw CER is dominated by these three items; normalisation, and the median, are not. The same measurements, aggregated three ways:

Table with columns: Aggregation (normalised), CER
Aggregation (normalised)	CER
Micro (reference)	11.3 %
Median per image	8.0 %
Macro (mean per image)	68.1 %

Human evaluation (3 Lingala-speaking validators, 25 test images, majority verdict):

Table with columns: Verdict, Share, 95 % CI
Verdict	Share	95 % CI
Correct	32 %	17–52 %
Correct or partially correct (recoverable)	92 %	75–98 %
Incorrect	8 %	—

Inter-rater agreement is α = 0.665, above the 0.60 threshold set before the campaign: this result is usable. The error validators cite most is "wrong ɔ/ɛ character" (43 mentions out of 50), ahead of invented text (21) and ordinary character errors (19). The automatic metric neutralises these letters; the validators do not.

Limitations

Looping outputs: about 3 in 50 outputs repeat until the length limit. Always bound max_new_tokens; a repetition check at inference removes the problem without retraining.
ɔ and ɛ remain the main difficulty, as the human evaluation shows.
Invalid UTF-8 bytes appear in place of "ɔ́" (open o with an accent) in about 1 output in 25: an interaction between the added tokens and combining accents.
Small test set: 50 images. Near 95 % accuracy, the uncertainty is about ±6 points; this set cannot prove or rule out a 95 % target.
Checkpoint chosen on the test set (see Training procedure).
Line-level only: trained on single text lines, not full pages.

Deployment

In the MALOBA inference gateway (LiteLLM, OpenAI-compatible), this model is served under the alias maloba-ocr, route POST /v1/chat/completions, with the image sent as a base64 image_url and the prompt above. The context must allow at least 4,096 tokens.

Licence

Model weights: Apache 2.0, the licence of Qwen/Qwen2-VL-2B-Instruct.
Training data: the licence of the MALOBA OCR dataset is being aligned on the Nwulite Obodo Open Data License (NOODL-1.0), already applied to the MALOBA text datasets. CDS asks users from high-income countries and commercial entities to publicly credit the MALOBA project (UNDP Republic of Congo — language digitalisation initiative) in any publication, product, model or output derived from this model. Contact: contact@congo-digital.com.

Citation

bibtex

@misc{cds2026qwenvllingala,  title        = {qwen-vl-lingala-qlora-vf: Lingala text recognition with Qwen2-VL},  author       = {{Congo Digital Services}},  year         = {2026},  howpublished = {\url{https://huggingface.co/Congo-digital-service/qwen-vl-lingala-qlora-vf}},  note         = {MALOBA project, UNDP Republic of Congo}}

Creators

Congo Digital Services (CDS SARL): https://www.congo-digital.com/
In collaboration with the MALOBA community (https://maloba.congo-digital.com/), SIL Congo (document collection), the Ministry of Posts, Telecommunications and the Digital Economy of the Republic of Congo and UNDP Congo.

Created in August 2026. Card updated on 30 September 2026.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

Congo-digital-service

Model Tree

Base

Qwen/Qwen2-VL-2B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer