osos3lom

receiptai

Model details

Base model: HuggingFaceTB/SmolVLM-256M-Instruct (Idefics3 architecture)
Method: LoRA, r=16, alpha=32, dropout=0.05
Adapted modules: 211 explicit fully-qualified Linear layers — text attention (120), text MLP (90) and the modality connector (1). The vision tower is not adapted. Targets are listed by full name rather than by suffix, because suffix matching on q/k/v/o_proj silently adapts the vision encoder as well.
Trainable parameters: 5,090,304 of 261,575,232 (1.95%)
Precision: FP16 with gradient checkpointing, trained on one RTX 2080 Ti (11 GB)

Intended use

In-browser receipt field extraction for bookkeeping, where documents must not leave the device. Not suitable for unattended accounting: see the measured accuracy below. Every extraction should be reviewed by a person before it reaches a ledger.

Training data

Table with columns: Source, Rows, Label provenance
Source	Rows	Label provenance
`synthetic_v2`	2,000	Generator-produced, exact by construction
`teacher_distilled`	687	Gemini-distilled from real CORU scans, QC-gated

2,275 train / 275 validation / 137 test, disjoint by image SHA-256. The 50-receipt gold evaluation set is excluded from all splits by hash.

Teacher labels passed two gates. Math reconciliation allows a bounded residual, because real receipts carry delivery and service lines the schema does not model (verified example: items 542.05 + delivery 15 + tax 77.99 = total 635.04). Grounding drops any numeric value that does not appear in the transcribed text: 175 subtotals, 145 VAT amounts and 15 totals were back-computed by the teacher rather than read, and were removed. vat_percent is dropped whenever vat_amount is, since the rate is not knowable without the amount.

Synthetic data spans 24 merchants, 5 VAT regimes (SAR 15/5/0%, AED 5%, EGP 14%), discounts, service charges, cash-rounding drift and refunds, rendered across varied widths, fonts and layouts, then augmented with perspective, rotation, uneven lighting, shadow, defocus, sensor noise, JPEG recompression and surface compositing.

Training procedure

Loss is computed only over the assistant's JSON answer; the prompt and image placeholder tokens are masked to -100. This is load-bearing: for a typical receipt the prefix is ~1,733 tokens of which ~1,088 are image placeholders, so an unmasked objective puts the large majority of loss mass on tokens the model must never predict.

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Epochs	3 (early stopping, patience 1)
Effective batch size	8 (per-device 1 × grad accum 8)
Learning rate	2e-4, cosine with 3% warmup
Gradient clipping	1.0
Optimizer	AdamW, weight decay 0.01
Seed	42
Supervised token fraction	32.3%

Table with columns: Epoch, Train loss, Val loss
Epoch	Train loss	Val loss
1	0.4598	0.2328
2	0.1919	0.1944
3	0.1672	0.1909

Runtime 158.9 minutes. Full configuration in training_run.json.

Evaluation

Two held-out sets. Fields with fewer than 5 gold examples are excluded from the gate score.

In-domain — 50 synthetic gold receipts

Table with columns: Field, base, v3
Field	base	v3
Store	0%	100%
Date	18%	100%
Total	4%	100%
Subtotal	2%	100%
VAT amount	2%	100%
Line-item F1	0.0	97.7
Gate score	4.7	99.3

Out-of-domain — 37 real held-out CORU scans

Table with columns: Field, base, synthetic-only, v3
Field	base	synthetic-only	v3
Store	8.1%	0%	29.7%
Date	0%	10.8%	46.0%
Total	11.1%	41.7%	63.9%
Subtotal	0%	32.0%	56.0%
VAT amount	8.0%	28.0%	68.0%
All money fields	0%	8.1%	32.4%
Line-item F1	0.0	0.0	5.3
Gate score	5.4	22.5	52.7

Reproduce with:

text

python data-pipeline/run_eval.py --arm adapter --checkpoint checkpoints/smolvlm-receipt-v3-combinedpython data-pipeline/eval_harness.py --gold data-pipeline/output/gold_eval/manifest.jsonl \    --pred data-pipeline/output/eval_runs/pred_adapter.jsonlpython data-pipeline/compare_arms.py --candidate v3_real

Limitations and risks

The in-domain score is mostly lookup, not reading. Every one of the 12 gold merchants and 21 gold line-item descriptions also appears verbatim in the training data, so store and line-item scores measure recall of a fixed catalogue. Random 15-digit tax numbers (96%) are the one in-domain field that cannot be memorized.
Roughly half of fields on a real receipt are correct, with wide error bars. Gate score 52.7, bootstrap 95% CI [42.9, 62.5] on n=37, measured as agreement with Gemini rather than against verified truth. This is a review aid, not an automation.
It invents subtotal and VAT. On the 12 real receipts where the reference has no printed subtotal or VAT, the model stated one on 11 (91.7%). It learned from synthetic data, where every receipt prints a VAT breakdown, that every receipt does. "Reference has none" comes from Gemini grounding, so human labels are needed to confirm the rate. The browser parser therefore cannot treat a stated subtotal or VAT as evidence that it was printed.
Line-item extraction does not work on real receipts (F1 5.3 against 97.7 in-domain). The gap is the clearest evidence of remaining renderer overfitting.
Tax numbers are unmeasured. Only 1 of 37 real receipts prints one; the field is excluded from the gate for lack of support.
The real evaluation set is teacher-referenced, not human-verified, so it carries the teacher's error rate. A human-verified set belongs at data-pipeline/output/coru_images/manual_gold.jsonl before any public accuracy claim.
Arabic merchant names are the weakest field at 29.7%, which matters because store identity drives expense categorization downstream.
Real receipts are drawn from CORU, which is predominantly Egyptian. Saudi ZATCA coverage comes mostly from synthetic data, so ZATCA-specific performance is less well evidenced than the name of the project suggests.

Next step if higher accuracy is needed

CORU ships 5,838 real bounding boxes that this pipeline does not use. A two-stage design — detect field regions from those boxes, then read each crop at native resolution — addresses both the line-item failure and the token-budget ceiling of a 256M model on dense bilingual text.

Framework versions

Training and evaluation: Transformers 4.57.6, PEFT 0.21.2, PyTorch 2.6.0+cu124. Pinned in data-pipeline/requirements.lock.txt. Installing the ONNX export tooling replaced the earlier Transformers 5.20.2 at 2026-10-23 23:59, before any v2/v3 run started. Re-running 6 gold receipts reproduced the saved completions token-for-token (greedy decoding).

💻 Quickstart Inference

1. Python (Transformers + PEFT)

python

import torchfrom PIL import Imagefrom transformers import AutoProcessor, AutoModelForVision2Seqfrom peft import PeftModel # 1. Load processor and base modelbase_model_id = "HuggingFaceTB/SmolVLM-256M-Instruct"adapter_id = "osos3lom/receiptai" processor = AutoProcessor.from_pretrained(adapter_id)model = AutoModelForVision2Seq.from_pretrained(    base_model_id,    torch_dtype=torch.float16,    device_map="auto")model = PeftModel.from_pretrained(model, adapter_id) # 2. Format receipt image and promptimage = Image.open("receipt.jpg")messages = [    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "Extract all receipt details as JSON."}        ]    }] prompt = processor.apply_chat_template(messages, add_generation_prompt=True)inputs = processor(text=prompt, images=[image], return_tensors="pt").to(model.device) # 3. Generate structured JSONwith torch.no_grad():    generated_ids = model.generate(**inputs, max_new_tokens=384)response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]print(response)

2. Browser WebGPU (Transformers.js v3)

This repository includes pre-quantized 4-bit ONNX weights (q4f16) for zero-server in-browser inference:

javascript

import { AutoProcessor, AutoModelForVision2Seq, RawImage } from "@huggingface/transformers"; const modelId = "osos3lom/receiptai"; // 1. Initialize processor & 4-bit quantized WebGPU modelconst processor = await AutoProcessor.from_pretrained(modelId);const model = await AutoModelForVision2Seq.from_pretrained(modelId, {  dtype: "q4f16",  device: "webgpu", // Automatic WASM fallback supported}); // 2. Load receipt imageconst image = await RawImage.fromURL("https://example.com/receipt.jpg"); // 3. Prepare promptconst messages = [  {    role: "user",    content: [      { type: "image" },      { type: "text", text: "Extract all receipt details as JSON." },    ],  },];const text = processor.apply_chat_template(messages, { add_generation_prompt: true });const inputs = await processor(text, image); // 4. Generate local structured extractionconst outputs = await model.generate({ ...inputs, max_new_tokens: 384 });const decoded = processor.batch_decode(outputs, { skip_special_tokens: true });console.log(decoded[0]);

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

osos3lom

Model Tree

Base

HuggingFaceTB/SmolVLM-256M-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

Model details

Base model: HuggingFaceTB/SmolVLM-256M-Instruct (Idefics3 architecture)
Method: LoRA, r=16, alpha=32, dropout=0.05
Adapted modules: 211 explicit fully-qualified Linear layers — text attention (120), text MLP (90) and the modality connector (1). The vision tower is not adapted. Targets are listed by full name rather than by suffix, because suffix matching on q/k/v/o_proj silently adapts the vision encoder as well.
Trainable parameters: 5,090,304 of 261,575,232 (1.95%)
Precision: FP16 with gradient checkpointing, trained on one RTX 2080 Ti (11 GB)

Intended use

Training data

Table with columns: Source, Rows, Label provenance
Source	Rows	Label provenance
`synthetic_v2`	2,000	Generator-produced, exact by construction
`teacher_distilled`	687	Gemini-distilled from real CORU scans, QC-gated

2,275 train / 275 validation / 137 test, disjoint by image SHA-256. The 50-receipt gold evaluation set is excluded from all splits by hash.

Training procedure

Table with columns: Hyperparameter, Value
Hyperparameter	Value
Epochs	3 (early stopping, patience 1)
Effective batch size	8 (per-device 1 × grad accum 8)
Learning rate	2e-4, cosine with 3% warmup
Gradient clipping	1.0
Optimizer	AdamW, weight decay 0.01
Seed	42
Supervised token fraction	32.3%

Table with columns: Epoch, Train loss, Val loss
Epoch	Train loss	Val loss
1	0.4598	0.2328
2	0.1919	0.1944
3	0.1672	0.1909

Runtime 158.9 minutes. Full configuration in training_run.json.

Evaluation

Two held-out sets. Fields with fewer than 5 gold examples are excluded from the gate score.

In-domain — 50 synthetic gold receipts

Table with columns: Field, base, v3
Field	base	v3
Store	0%	100%
Date	18%	100%
Total	4%	100%
Subtotal	2%	100%
VAT amount	2%	100%
Line-item F1	0.0	97.7
Gate score	4.7	99.3

Out-of-domain — 37 real held-out CORU scans

Table with columns: Field, base, synthetic-only, v3
Field	base	synthetic-only	v3
Store	8.1%	0%	29.7%
Date	0%	10.8%	46.0%
Total	11.1%	41.7%	63.9%
Subtotal	0%	32.0%	56.0%
VAT amount	8.0%	28.0%	68.0%
All money fields	0%	8.1%	32.4%
Line-item F1	0.0	0.0	5.3
Gate score	5.4	22.5	52.7

Reproduce with:

text

python data-pipeline/run_eval.py --arm adapter --checkpoint checkpoints/smolvlm-receipt-v3-combinedpython data-pipeline/eval_harness.py --gold data-pipeline/output/gold_eval/manifest.jsonl \    --pred data-pipeline/output/eval_runs/pred_adapter.jsonlpython data-pipeline/compare_arms.py --candidate v3_real

python data-pipeline/run_eval.py --arm adapter --checkpoint checkpoints/smolvlm-receipt-v3-combinedpython data-pipeline/eval_harness.py --gold data-pipeline/output/gold_eval/manifest.jsonl \    --pred data-pipeline/output/eval_runs/pred_adapter.jsonlpython data-pipeline/compare_arms.py --candidate v3_real

Limitations and risks

The in-domain score is mostly lookup, not reading. Every one of the 12 gold merchants and 21 gold line-item descriptions also appears verbatim in the training data, so store and line-item scores measure recall of a fixed catalogue. Random 15-digit tax numbers (96%) are the one in-domain field that cannot be memorized.
Roughly half of fields on a real receipt are correct, with wide error bars. Gate score 52.7, bootstrap 95% CI [42.9, 62.5] on n=37, measured as agreement with Gemini rather than against verified truth. This is a review aid, not an automation.
It invents subtotal and VAT. On the 12 real receipts where the reference has no printed subtotal or VAT, the model stated one on 11 (91.7%). It learned from synthetic data, where every receipt prints a VAT breakdown, that every receipt does. "Reference has none" comes from Gemini grounding, so human labels are needed to confirm the rate. The browser parser therefore cannot treat a stated subtotal or VAT as evidence that it was printed.
Line-item extraction does not work on real receipts (F1 5.3 against 97.7 in-domain). The gap is the clearest evidence of remaining renderer overfitting.
Tax numbers are unmeasured. Only 1 of 37 real receipts prints one; the field is excluded from the gate for lack of support.
The real evaluation set is teacher-referenced, not human-verified, so it carries the teacher's error rate. A human-verified set belongs at data-pipeline/output/coru_images/manual_gold.jsonl before any public accuracy claim.
Arabic merchant names are the weakest field at 29.7%, which matters because store identity drives expense categorization downstream.
Real receipts are drawn from CORU, which is predominantly Egyptian. Saudi ZATCA coverage comes mostly from synthetic data, so ZATCA-specific performance is less well evidenced than the name of the project suggests.

Next step if higher accuracy is needed

Framework versions

Training and evaluation: Transformers 4.57.6, PEFT 0.21.2, PyTorch 2.6.0+cu124. Pinned in data-pipeline/requirements.lock.txt. Installing the ONNX export tooling replaced the earlier Transformers 5.20.2 at 2026-10-23 23:59, before any v2/v3 run started. Re-running 6 gold receipts reproduced the saved completions token-for-token (greedy decoding).

💻 Quickstart Inference

1. Python (Transformers + PEFT)

python

import torchfrom PIL import Imagefrom transformers import AutoProcessor, AutoModelForVision2Seqfrom peft import PeftModel # 1. Load processor and base modelbase_model_id = "HuggingFaceTB/SmolVLM-256M-Instruct"adapter_id = "osos3lom/receiptai" processor = AutoProcessor.from_pretrained(adapter_id)model = AutoModelForVision2Seq.from_pretrained(    base_model_id,    torch_dtype=torch.float16,    device_map="auto")model = PeftModel.from_pretrained(model, adapter_id) # 2. Format receipt image and promptimage = Image.open("receipt.jpg")messages = [    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "Extract all receipt details as JSON."}        ]    }] prompt = processor.apply_chat_template(messages, add_generation_prompt=True)inputs = processor(text=prompt, images=[image], return_tensors="pt").to(model.device) # 3. Generate structured JSONwith torch.no_grad():    generated_ids = model.generate(**inputs, max_new_tokens=384)response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]print(response)

import torchfrom PIL import Imagefrom transformers import AutoProcessor, AutoModelForVision2Seqfrom peft import PeftModel # 1. Load processor and base modelbase_model_id = "HuggingFaceTB/SmolVLM-256M-Instruct"adapter_id = "osos3lom/receiptai" processor = AutoProcessor.from_pretrained(adapter_id)model = AutoModelForVision2Seq.from_pretrained(    base_model_id,    torch_dtype=torch.float16,    device_map="auto")model = PeftModel.from_pretrained(model, adapter_id) # 2. Format receipt image and promptimage = Image.open("receipt.jpg")messages = [    {        "role": "user",        "content": [            {"type": "image"},            {"type": "text", "text": "Extract all receipt details as JSON."}        ]    }] prompt = processor.apply_chat_template(messages, add_generation_prompt=True)inputs = processor(text=prompt, images=[image], return_tensors="pt").to(model.device) # 3. Generate structured JSONwith torch.no_grad():    generated_ids = model.generate(**inputs, max_new_tokens=384)response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]print(response)

2. Browser WebGPU (Transformers.js v3)

This repository includes pre-quantized 4-bit ONNX weights (q4f16) for zero-server in-browser inference:

javascript

import { AutoProcessor, AutoModelForVision2Seq, RawImage } from "@huggingface/transformers"; const modelId = "osos3lom/receiptai"; // 1. Initialize processor & 4-bit quantized WebGPU modelconst processor = await AutoProcessor.from_pretrained(modelId);const model = await AutoModelForVision2Seq.from_pretrained(modelId, {  dtype: "q4f16",  device: "webgpu", // Automatic WASM fallback supported}); // 2. Load receipt imageconst image = await RawImage.fromURL("https://example.com/receipt.jpg"); // 3. Prepare promptconst messages = [  {    role: "user",    content: [      { type: "image" },      { type: "text", text: "Extract all receipt details as JSON." },    ],  },];const text = processor.apply_chat_template(messages, { add_generation_prompt: true });const inputs = await processor(text, image); // 4. Generate local structured extractionconst outputs = await model.generate({ ...inputs, max_new_tokens: 384 });const decoded = processor.batch_decode(outputs, { skip_special_tokens: true });console.log(decoded[0]);

import { AutoProcessor, AutoModelForVision2Seq, RawImage } from "@huggingface/transformers"; const modelId = "osos3lom/receiptai"; // 1. Initialize processor & 4-bit quantized WebGPU modelconst processor = await AutoProcessor.from_pretrained(modelId);const model = await AutoModelForVision2Seq.from_pretrained(modelId, {  dtype: "q4f16",  device: "webgpu", // Automatic WASM fallback supported}); // 2. Load receipt imageconst image = await RawImage.fromURL("https://example.com/receipt.jpg"); // 3. Prepare promptconst messages = [  {    role: "user",    content: [      { type: "image" },      { type: "text", text: "Extract all receipt details as JSON." },    ],  },];const text = processor.apply_chat_template(messages, { add_generation_prompt: true });const inputs = await processor(text, image); // 4. Generate local structured extractionconst outputs = await model.generate({ ...inputs, max_new_tokens: 384 });const decoded = processor.batch_decode(outputs, { skip_special_tokens: true });console.log(decoded[0]);

receiptai

README

Model details

Intended use

Training data

Training procedure

Evaluation

Limitations and risks

Next step if higher accuracy is needed

Framework versions

💻 Quickstart Inference

1. Python (Transformers + PEFT)

2. Browser WebGPU (Transformers.js v3)

Explore FriendliAI today

README

Model details

Intended use

Training data

Training procedure

Evaluation

Limitations and risks

Next step if higher accuracy is needed

Framework versions

💻 Quickstart Inference

1. Python (Transformers + PEFT)

2. Browser WebGPU (Transformers.js v3)