Highlights
MIST-Extract-1.0 turns a document into JSON. Give it a JSON template and a document (text, a photo or scan, or
both), and it fills in the template. Each field it returns comes with a calibrated confidence. It is a 4B model,
built for English and African-language documents. For a faster 2B version, see
MIST-Extract-Flash-1.0.
- English documents, extracted end to end: on 486 held-out English documents (receipts, notices, emails,
contracts and more), it gets 91.8% of documents completely right. NuExtract3, built on the same 4B base, gets
67.9%. On English receipt photos it gets every field right.
- Strongest model we tested on African languages. On named-entity extraction in 21 African languages (MasakhaNER
2.0), it scores 0.728 field accuracy. NuExtract3 scores 0.393, Qwen3.5-4B 0.432 and GLiNER2.5-multi 0.452.
- Ahead of NuExtract3 on every part of our held-out set, general documents included (0.941 vs 0.922), at the
same size and slightly faster (6.0 vs 5.6 documents per second).
- Calibrated confidence for every field. When it says 0.66, it is right about 66% of the time (calibration error
0.008 on held-out fields), so you can send uncertain values to a person and keep the rest automatic.
- Same template format as NuExtract, so templates move between the two without changes.
What it's for
MIST-Extract is built for the everyday documents of African businesses and their customers, in English and African
languages:
- Mobile money and banking: transfer confirmations, bank and wallet SMS, alerts and statements.
- Receipts and bills: shop, market, utility and prepaid receipts in naira, cedis, shillings, rand and other
African currencies, as text, photos or screenshots.
- Letters and forms: tenancy and school letters, notices, IDs, tickets, complaints and claims.
- Names in 20+ African languages: people, organisations, places and dates.
On these documents it is more accurate than the general-purpose extractors we tested, and its confidence tells you
which fields to check. For general Western invoices, multi-page PDFs and foreign receipts, use a general-purpose
extractor such as NuExtract3.
Model Overview
Table | |
|---|
| Base | Qwen/Qwen3.5-4B-Base (Apache-2.0), full fine-tune, vision encoder frozen |
| Input | a JSON template, a document as text and/or one image, optional instructions |
| Output | JSON with the template's shape, plus a confidence for each field |
| Precision | bf16 |
| Context | 262,144-token window from the Qwen3.5 base; we serve it at 131,072. Images are scaled to about 1.3 megapixels |
| Smaller sibling | MIST-Extract-Flash-1.0 (2B), 1.6× faster |
The template has the shape of the answer. Each leaf names a type:
Table with columns: Type, Meaning| Type | Meaning |
|---|
"verbatim-string" | copied exactly from the document |
"string" | free text, may be normalised or paraphrased |
"number", "integer" | plain numbers, no currency symbols or thousands separators |
"date" | ISO YYYY-MM-DD |
"date-time" | ISO date and time |
"boolean" | true or false |
["a", "b"] | pick one option |
[["a", "b"]] | pick any number of options |
["string"] | a list of values |
[{...}] | a list of objects |
A missing value comes back as null, and an empty list as [].
Quickstart
pip install "transformers>=5.17" torch pillow huggingface_hub
import sysfrom huggingface_hub import snapshot_download path = snapshot_download("olaverse/MIST-Extract-1.0")sys.path.insert(0, path) # the repo ships mist_extract.pyfrom mist_extract import Extractor x = Extractor.from_pretrained(path, device="cuda") template = { "merchant": "verbatim-string", "date": "date", "total": "number", "payment_method": ["cash", "card", "mobile money", "bank transfer"], "items": [{"name": "verbatim-string", "quantity": "integer", "price": "number"}],} # from a photo or scanout = x.extract(template, image="receipt.jpg") # or from textout = x.extract(template, text="Paid 2,500 naira at Mama Put on 3 May 2024 with my card. 2 plates of jollof.") print(out["data"]) # {"merchant": "Mama Put", "date": "2024-05-03", "total": 2500, ...}print(out["confidence"]) # same shape, a number in [0, 1] for each field
For many documents, x.extract_batch([{"template": ..., "text": ..., "image": ...}, ...]) runs them in one batch.
Add instructions="..." to steer formats or definitions, for example "amounts in kobo".
Offline batch inference (vLLM)
The model is a standard Qwen3_5ForConditionalGeneration checkpoint, so vLLM serves it as is. Build the prompt below,
pass the image (if any) as multi_modal_data, and decode greedily. Field confidences need token probabilities,
which the Extractor above computes for you.
import jsonfrom PIL import Imagefrom vllm import LLM, SamplingParams SYSTEM = "You extract information from documents into JSON that matches the template." def prompt(template, text="", instructions="", image=False): parts = [f"# Template:\n{json.dumps(template, ensure_ascii=False)}"] if instructions: parts.append(f"# Instructions:\n{instructions}") doc = ("<|vision_start|><|image_pad|><|vision_end|>\n" if image else "") + text parts.append(f"# Document:\n{doc}") return (f"<|im_start|>system\n{SYSTEM}<|im_end|>\n<|im_start|>user\n" + "\n".join(parts) + "<|im_end|>\n<|im_start|>assistant\n") def fit(img, max_tokens=1280): """Downscale to at most 1,280 vision tokens (one token per 32 x 32 pixels), as in training.""" w, h = img.size if w * h > max_tokens * 1024: s = (max_tokens * 1024 / (w * h)) ** 0.5 * 0.97 img = img.resize((int(w * s), int(h * s))) return img def main(): llm = LLM("olaverse/MIST-Extract-1.0", max_model_len=131072, limit_mm_per_prompt={"image": 1}) params = SamplingParams(temperature=0, max_tokens=4096) template = {"merchant": "verbatim-string", "date": "date", "total": "number"} requests = [ {"prompt": prompt(template, text="Paid 2,500 naira at Mama Put on 3 May 2024.")}, {"prompt": prompt(template, image=True), "multi_modal_data": {"image": fit(Image.open("receipt.jpg").convert("RGB"))}}, ] for out in llm.generate(requests, params): # one batched call for any number of documents print(json.loads(out.outputs[0].text)) if __name__ == "__main__": # vLLM starts worker processes; the guard is required main()
OpenAI-compatible server (vLLM)
vllm serve olaverse/MIST-Extract-1.0 \ --limit-mm-per-prompt '{"image": 1, "video": 0}' \ --chat-template-content-format openai \ --max-model-len 131072
The model's chat template builds the extraction prompt. Pass the template (and optional instructions) as
chat_template_kwargs, and put the document in the user message: text, one image, or both. Send the template as
compact JSON, as in training, and use temperature=0. Large photos are downscaled automatically to the training size
(about 1.3 megapixels).
Text
import jsonfrom openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") template = { "store": "verbatim-string", "date": "date", "total": "number", "currency": ["NGN", "GHS", "KES", "ZAR", "USD", "Other"], "items": [{"name": "verbatim-string", "quantity": "integer", "price": "number"}],} response = client.chat.completions.create( model="olaverse/MIST-Extract-1.0", temperature=0, messages=[{ "role": "user", "content": [{"type": "text", "text": "Yesterday (12 March 2025) I paid ₦7,500 at Mama Put Kitchen, Yaba: " "2 plates of jollof rice at ₦3,000 each and a bottle of Zobo for ₦1,500."}], }], extra_body={"chat_template_kwargs": {"template": json.dumps(template)}},)print(response.choices[0].message.content)
Output:
{"store": "Mama Put Kitchen, Yaba", "date": "2025-03-12", "total": 7500, "currency": "NGN", "items": [{"name": "jollof rice", "quantity": 2, "price": 3000}, {"name": "Zobo", "quantity": 1, "price": 1500}]}
Image
import base64, jsonfrom openai import OpenAI client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1") with open("receipt.jpg", "rb") as f: data_url = "data:image/jpeg;base64," + base64.b64encode(f.read()).decode("utf-8") template = {"store": "verbatim-string", "date": "date", "total": "number", "tax": "number"} response = client.chat.completions.create( model="olaverse/MIST-Extract-1.0", temperature=0, messages=[{"role": "user", "content": [{"type": "image_url", "image_url": {"url": data_url}}]}], extra_body={"chat_template_kwargs": {"template": json.dumps(template)}},)print(response.choices[0].message.content)
Instructions
response = client.chat.completions.create( model="olaverse/MIST-Extract-1.0", temperature=0, messages=[{"role": "user", "content": "Your MTN MoMo transfer of GHS 250.00 to KOFI MENSAH was successful. " "Fee GHS 2.50. Ref 88213374. 14/10/2025"}], extra_body={"chat_template_kwargs": { "template": json.dumps({"recipient": "verbatim-string", "amount": "number", "fee": "number", "reference": "verbatim-string", "date": "date"}), "instructions": "Dates in this message are day/month/year.", }},)print(response.choices[0].message.content)# {"recipient": "KOFI MENSAH", "amount": 250, "fee": 2.5, "reference": "88213374", "date": "2025-10-14"}
The server returns the JSON only. For per-field confidences, use the Extractor from the Quickstart.
Confidence
A field's confidence is the lowest probability among the tokens that spell its value, put through a calibration
(Platt scaling) fitted on held-out fields and stored in mist_extract_config.json. On the held-out half not used for
fitting, calibration error (ECE) falls from 0.045 to 0.008:
Table with columns: Calibrated confidence, Fields, Share actually right| Calibrated confidence | Fields | Share actually right |
|---|
| below 0.5 | 356 | 40% |
| 0.5 to 0.8 | 694 | 66% |
| 0.8 to 0.95 | 916 | 89% |
| 0.95 to 0.99 | 1,917 | 98% |
| 0.99 and above | 2,859 | 99.9% |
Route fields below a threshold you choose to review. An empty list ([]) has no value tokens, so it has no
confidence.
Evaluation
Every model saw the same documents (text, or the same image), the same template and instructions, greedy decoding
and up to 2,048 new tokens, in the prompt format its makers document. All answers are scored by the same rules:
- Facts are scored exactly (numbers, dates, IDs, choices), after parsing numbers and dates written in any format.
- Free text is scored by word overlap, with a Jaccard index of at least 0.5.
- Lists are compared as sets.
Field accuracy is the share of fields right. Documents fully right is the share of documents with every
field right.
Table with columns: Benchmark, Documents, What it is| Benchmark | Documents | What it is |
|---|
| Olaverse held-out | 2,016 | Held out from our training data, so in-domain: receipt photos, African documents, general documents, and MasakhaNER 2.0 test passages (about 50 per language). 32 languages. |
| CORD-v2 test | 100 | Indonesian receipt photos. We trained on CORD's training split. |
| nanonets KIE | 987 | Malaysian receipt photos, not used in training. |
| Invoice Extraction Benchmark v1 | 181 | Invoices: PDFs (given as text), phone photos and scans. Not used in training. |
Models compared: MIST-Extract-1.0 (4B), MIST-Extract-Flash-1.0 (2B), NuExtract3
(4B), Qwen3.5-4B without fine-tuning,
LFM2-1.2B-Extract for text with
LFM2.5-VL-1.6B-Extract for images, and
GLiNER2.5-multi. GLiNER2.5-multi extracts spans only, so it is
scored on the named-entity lists only.
English documents (Olaverse held-out, 486 documents)
Table with columns: MIST-Extract (4B), MIST-Extract-Flash (2B), NuExtract3 (4B), Qwen3.5-4B, LFM2-Extract | MIST-Extract (4B) | MIST-Extract-Flash (2B) | NuExtract3 (4B) | Qwen3.5-4B | LFM2-Extract |
|---|
| Documents fully right | 91.8% | 90.5% | 67.9% | 34.0% | 4.5% |
| Field accuracy, receipt photos (327) | 1.000 | 0.999 | 0.925 | 0.738 | 0.565 |
| Field accuracy, text documents (159) | 0.965 | 0.956 | 0.967 | 0.940 | 0.700 |
Our focus: field accuracy
Table with columns: MIST-Extract (4B), MIST-Extract-Flash (2B), NuExtract3 (4B), Qwen3.5-4B, LFM2-Extract, GLiNER2.5-multi | MIST-Extract (4B) | MIST-Extract-Flash (2B) | NuExtract3 (4B) | Qwen3.5-4B | LFM2-Extract | GLiNER2.5-multi |
|---|
| Olaverse held-out (2,016 documents) | 0.898 | 0.879 | 0.801 | 0.762 | 0.432 | — |
| ↳ MasakhaNER 2.0, 21 African languages | 0.728 | 0.701 | 0.393 | 0.432 | 0.003 | 0.452 |
| ↳ receipt photos | 1.000 | 0.999 | 0.925 | 0.738 | 0.565 | — |
| ↳ African documents | 0.931 | 0.912 | 0.919 | 0.890 | 0.556 | — |
| ↳ general documents | 0.941 | 0.900 | 0.922 | 0.912 | 0.553 | — |
| CORD-v2 receipts (we trained on its train split) | 0.899 | 0.921 | 0.672 | 0.619 | 0.282 | — |
The fields that matter most
Dates, amounts and currencies on the Olaverse held-out documents (receipts, mobile money, bills, letters, forms):
Table with columns: Field type, MIST-Extract (4B), MIST-Extract-Flash (2B), NuExtract3 (4B), Qwen3.5-4B| Field type | MIST-Extract (4B) | MIST-Extract-Flash (2B) | NuExtract3 (4B) | Qwen3.5-4B |
|---|
| Dates (792 fields) | 0.979 | 0.975 | 0.884 | 0.833 |
| Amounts and numbers (1,757) | 0.984 | 0.970 | 0.964 | 0.919 |
| Currency (119) | 0.975 | 0.975 | 0.966 | 0.403 |
Documents fully right
Table with columns: MIST-Extract (4B), MIST-Extract-Flash (2B), NuExtract3 (4B), Qwen3.5-4B, LFM2-Extract | MIST-Extract (4B) | MIST-Extract-Flash (2B) | NuExtract3 (4B) | Qwen3.5-4B | LFM2-Extract |
|---|
| Olaverse held-out | 62.4% | 57.6% | 37.3% | 29.4% | 1.4% |
| CORD-v2 receipts | 64.0% | 71.0% | 30.0% | 25.0% | 5.0% |
Outside our focus: general Western receipts and invoices
These are not the documents MIST-Extract was built for, and a general-purpose extractor does better on them. They are
reported for completeness:
Table with columns: MIST-Extract (4B), MIST-Extract-Flash (2B), NuExtract3 (4B), Qwen3.5-4B, LFM2-Extract | MIST-Extract (4B) | MIST-Extract-Flash (2B) | NuExtract3 (4B) | Qwen3.5-4B | LFM2-Extract |
|---|
| nanonets KIE (987 Malaysian receipt photos), field accuracy | 0.819 | 0.759 | 0.917 | 0.899 | 0.654 |
| Invoice Extraction Benchmark (181 invoices), field accuracy | 0.793 | 0.759 | 0.907 | 0.880 | 0.451 |
Valid JSON
Table with columns: Benchmark, MIST-Extract (4B), MIST-Extract-Flash, NuExtract3, Qwen3.5-4B, LFM2-Extract| Benchmark | MIST-Extract (4B) | MIST-Extract-Flash | NuExtract3 | Qwen3.5-4B | LFM2-Extract |
|---|
| Olaverse held-out | 100% | 100% | 100% | 99.9% | 94.1% |
| Invoice Extraction Benchmark (outside our focus) | 94.5% | 96.1% | 96.7% | 95.6% | 80.7% |
Speed
Documents per second on one RTX A6000 (48 GB) under vLLM 0.31, greedy decoding, all 3,284 benchmark documents
(GLiNER2.5-multi: its 1,000 entity documents).
Table with columns: MIST-Extract (4B), MIST-Extract-Flash, NuExtract3, Qwen3.5-4B, LFM2-Extract, GLiNER2.5-multi| MIST-Extract (4B) | MIST-Extract-Flash | NuExtract3 | Qwen3.5-4B | LFM2-Extract | GLiNER2.5-multi |
|---|
| 6.0 | 9.7 | 5.6 | 5.4 | 7.7 | 18.4 |
Supported languages
The template keys stay in English; the document can be in any of the languages below. Field accuracy is on the
Olaverse held-out set, by document language.
- Full support: documents of every kind, many training examples. English, Naija (Nigerian Pidgin), isiZulu,
Kiswahili, Hausa, Yorùbá, Igbo, Somali and Amharic.
- Named entities: names of people, organisations, places and dates, from MasakhaNER 2.0 (600 training passages
each). Twi, chiShona, Fon, Chichewa, Bambara, Ghomálá', Mooré, Setswana, Kinyarwanda, Wolof, Ewe, isiXhosa,
Luganda and Luo. Other document types in these languages work less reliably.
- Light coverage: Indonesian (CORD receipts), Chinese, Arabic, Hindi, Russian, Spanish, French, German, Turkish
and Portuguese, with 15 to 900 training examples each. These come largely from the Qwen3.5 base; the test sets are
too small to report (1 to 6 documents).
Table with columns: Language, Documents, MIST-Extract (4B), MIST-Extract-Flash, NuExtract3, Qwen3.5-4B, GLiNER2.5-multi| Language | Documents | MIST-Extract (4B) | MIST-Extract-Flash | NuExtract3 | Qwen3.5-4B | GLiNER2.5-multi |
|---|
| English | 486 | 0.985 | 0.981 | 0.943 | 0.824 | — |
| Naija (Nigerian Pidgin) | 111 | 0.920 | 0.908 | 0.885 | 0.844 | 0.420 |
| isiZulu | 113 | 0.901 | 0.850 | 0.810 | 0.777 | 0.541 |
| Kiswahili | 109 | 0.889 | 0.864 | 0.845 | 0.818 | 0.438 |
| Hausa | 109 | 0.866 | 0.851 | 0.794 | 0.786 | 0.293 |
| Yorùbá | 107 | 0.866 | 0.829 | 0.751 | 0.757 | 0.594 |
| Igbo | 106 | 0.899 | 0.866 | 0.789 | 0.787 | 0.427 |
| Somali | 78 | 0.929 | 0.923 | 0.931 | 0.910 | — |
| Amharic | 75 | 0.893 | 0.859 | 0.870 | 0.846 | — |
| Twi | 50 | 0.839 | 0.863 | 0.645 | 0.677 | 0.645 |
| chiShona | 50 | 0.835 | 0.795 | 0.291 | 0.276 | 0.339 |
| Fon | 50 | 0.800 | 0.791 | 0.443 | 0.513 | 0.513 |
| Chichewa | 50 | 0.770 | 0.746 | 0.418 | 0.459 | 0.451 |
| Kinyarwanda | 50 | 0.750 | 0.692 | 0.433 | 0.375 | 0.300 |
| Bambara | 50 | 0.748 | 0.732 | 0.386 | 0.512 | 0.512 |
| Mooré | 50 | 0.740 | 0.707 | 0.431 | 0.593 | 0.577 |
| Ghomálá' | 50 | 0.726 | 0.726 | 0.425 | 0.540 | 0.637 |
| Wolof | 50 | 0.720 | 0.678 | 0.347 | 0.432 | 0.432 |
| Setswana | 50 | 0.698 | 0.713 | 0.450 | 0.457 | 0.465 |
| Luganda | 50 | 0.667 | 0.576 | 0.356 | 0.348 | 0.402 |
| Ewe | 50 | 0.658 | 0.650 | 0.385 | 0.410 | 0.325 |
| isiXhosa | 50 | 0.650 | 0.585 | 0.260 | 0.333 | 0.431 |
| Luo | 50 | 0.610 | 0.542 | 0.195 | 0.203 | 0.297 |
GLiNER2.5-multi is scored on the named-entity lists only. Entity lists are scored strictly: a list counts as right
only if it holds exactly the right names.
Training
- Method: full fine-tune of Qwen3.5-4B-Base for 2 epochs on 44,145 examples, 16,573 of them with an image. The
loss covered only the answer, and the vision encoder was frozen. Learning rate 1.5e-5, 8-bit AdamW, effective
batch 16, bf16, on one A100 80 GB.
- Public data:
- MasakhaNER 2.0 (AFL-3.0): named entities in 20 African
languages, capped at 600 passages per language.
- CORD-v2 (CC-BY-4.0): 900 receipt photos.
- Rendered receipts and invoices: 12,000 documents in African currencies and layouts. The renderer knows every
printed value, so the answers are exact.
- Generated documents: African documents (mobile-money and bank SMS, utility bills, IDs and forms, tenancy and
school letters, tickets) and general documents (emails, contracts, news, reports, listings), with phone-screenshot
versions. Each answer was written by one model and checked by a second, and fields the two disagreed on were
removed.
- Templates: key names vary and fields are randomly sub-selected, so the model follows the template rather than
memorising one.
Limitations
- Scope. Built for African and English everyday documents (see What it's for). On general
Western invoices and foreign receipts, dates, decimal amounts and currency codes go wrong more often; use a
general-purpose extractor for those.
- One image per call. For a multi-page scan, send the pages as text or extract page by page.
- Entity lists are copied as written, so spelling variants of a name count as different values.
Citation
@misc{mist-extract-2026, title = {MIST-Extract: document extraction for English and African languages, with field confidence}, author = {Olaverse}, year = {2026}, url = {https://huggingface.co/olaverse/MIST-Extract-1.0}}