LightOnOCR-3-4B-W4A16-AutoRound-GPTQ API & Inference Endpoint

Quantization Details

Table with columns: Parameter, Configuration, Details
Parameter	Configuration	Details
Quantization Scheme	`W4A16`	Weights INT4, Activations BF16
Group Size	`32`	Enhanced accuracy per weight block
Symmetry	`Symmetric` (`sym=True`)	Standard high-throughput configuration
Iterations	`1200`	Production-grade fine-grained optimization
Calibration Samples	`512` samples (`seqlen=4096`)	Broad domain calibration coverage
Vision Preservation	`quant_nontext_module=False`	Vision tower & multimodal projector kept in BF16 to prevent OCR/grounding degradation

Sibling Repositories & Formats

Choose the backend format suited to your inference engine:

AutoRound Native / Transformers: Vishva007/LightOnOCR-3-4B-W4A16-AutoRound
AutoGPTQ / Marlin: Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-GPTQ
vLLM / Compressed Tensors: Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-LLM-Compressor

Operating Modes

LightOnOCR-3 operates in two primary modes:

Plain Transcription Mode (prompt = ""): Transcribes text directly to Markdown layout without bounding boxes.
Grounding Mode (prompt = "grounding"): Outputs normalized bounding boxes [0, 1000] and semantic labels (text, title, table, image, chart, etc.) for each document component.

Quickstart

1. Python Inference via Transformers

bash

pip install --upgrade "transformers>=5.5.4" auto-round pillow torchvision

python

import torchfrom transformers import AutoProcessor, AutoModelForImageTextToText # Swap repo_id with your target variant:model_id = "Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-GPTQ" processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(    model_id,    torch_dtype=torch.bfloat16,    device_map="auto") # Set image URL or local pathimage_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" # Use "grounding" or "" (for transcription only)prompt = "grounding" content = [{"type": "image", "url": image_url}]if prompt:    content.append({"type": "text", "text": prompt}) conversation = [{"role": "user", "content": content}] inputs = processor.apply_chat_template(    conversation,    add_generation_prompt=True,    tokenize=True,    return_dict=True,    return_tensors="pt",    enable_thinking=False,  # LightOnOCR is optimized with thinking disabled).to(model.device) with torch.inference_mode():    output_ids = model.generate(**inputs, max_new_tokens=2048, do_sample=False) generated_ids = output_ids[0, inputs["input_ids"].shape[1]:]print(processor.decode(generated_ids, skip_special_tokens=True))

2. High-Throughput Serving with vLLM

(Recommended using Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-LLM-Compressor or Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-GPTQ)

Start Server

bash

vllm serve Vishva007/Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-GPTQ \  --trust-remote-code \  --dtype bfloat16 \  --max-model-len 8192 \  --default-chat-template-kwargs '{"enable_thinking": false}'

Query Server

python

import requests url = "http://localhost:8000/v1/chat/completions"image_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" payload = {    "model": "Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-GPTQ",    "messages": [        {            "role": "user",            "content": [                {"type": "image_url", "image_url": {"url": image_url}},                {"type": "text", "text": "grounding"}            ]        }    ],    "max_tokens": 2048,    "temperature": 0.0} response = requests.post(url, json=payload)print(response.json()["choices"][0]["message"]["content"])

Best Practices for Document Preprocessing

Resolution: Render scanned documents and PDFs at 400 DPI using a target longest dimension of 2048px.
Aspect Ratio: Maintain the native aspect ratio when resizing; do not stretch images, as bounding box coordinate accuracy depends on geometric integrity.
Thinking Tokens: Ensure enable_thinking=False is preserved in chat templates during generation.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5-$ 500 in free credits when you add your first $10.

PyTorch 2.14

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`

PyTorch 2.13

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`

PyTorch 2.12

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`

Citation & Acknowledgements

Quantized using Intel AutoRound. Base model created and released by LightOn AI.

bibtex

@misc{lightonocr2_2026,  title        = {LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR},  author       = {Said Taghadouini and Adrien Cavaill\`{e}s and Baptiste Aubertin},  year         = {2026},  howpublished = {\url{[https://arxiv.org/abs/2601.14251](https://arxiv.org/abs/2601.14251)}}} @article{cheng2023optimize,  title={Optimize Weight Rounding via Signed Gradient Descent for Large Language Model Quantization},  author={Cheng, Wenhua and Hai, Wei and Shen, Haihao and others},  journal={arXiv preprint arXiv:2309.05516},  year={2023}}

Parameter

Configuration

Details

Quantization Scheme

W4A16

Weights INT4, Activations BF16

Group Size

32

Enhanced accuracy per weight block

Symmetry

Symmetric (sym=True)

Standard high-throughput configuration

Iterations

1200

Production-grade fine-grained optimization

Calibration Samples

512 samples (seqlen=4096)

Broad domain calibration coverage

Vision Preservation

quant_nontext_module=False

Vision tower & multimodal projector kept in BF16 to prevent OCR/grounding degradation

import torchfrom transformers import AutoProcessor, AutoModelForImageTextToText # Swap repo_id with your target variant:model_id = "Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-GPTQ" processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained( model_id, torch_dtype=torch.bfloat16, device_map="auto") # Set image URL or local pathimage_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" # Use "grounding" or "" (for transcription only)prompt = "grounding" content = [{"type": "image", "url": image_url}]if prompt: content.append({"type": "text", "text": prompt}) conversation = [{"role": "user", "content": content}] inputs = processor.apply_chat_template( conversation, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", enable_thinking=False, # LightOnOCR is optimized with thinking disabled).to(model.device) with torch.inference_mode(): output_ids = model.generate(**inputs, max_new_tokens=2048, do_sample=False) generated_ids = output_ids[0, inputs["input_ids"].shape[1]:]print(processor.decode(generated_ids, skip_special_tokens=True))

import requests url = "http://localhost:8000/v1/chat/completions"image_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" payload = { "model": "Vishva007/LightOnOCR-3-4B-W4A16-AutoRound-GPTQ", "messages": [ { "role": "user", "content": [ {"type": "image_url", "image_url": {"url": image_url}}, {"type": "text", "text": "grounding"} ] } ], "max_tokens": 2048, "temperature": 0.0} response = requests.post(url, json=payload)print(response.json()["choices"][0]["message"]["content"])

Template

CUDA Version

Docker Image

Template ID

Deploy

PyTorch 2.14 (CUDA 12.6)

12.6

vishva123/cuda-12.6-pytorch-2.14-runpod

d7lxsa4w9m

PyTorch 2.14 (CUDA 13.0)

13.0

vishva123/cuda-13.0-pytorch-2.14-runpod

Template

CUDA Version

Docker Image

Template ID

Deploy

PyTorch 2.13 (CUDA 12.6)

12.6

vishva123/cuda-12.6-pytorch-2.13-runpod

gmlupxnxfk

PyTorch 2.13 (CUDA 13.0)

13.0

vishva123/cuda-13.0-pytorch-2.13-runpod

Template

CUDA Version

Docker Image

Template ID

Deploy

PyTorch 2.12 (CUDA 12.6)

12.6

vishva123/cuda-12.6-pytorch-2.12-runpod

ctmz86zmf0

PyTorch 2.12 (CUDA 13.0)

13.0

vishva123/cuda-13.0-pytorch-2.12-runpod

@misc{lightonocr2_2026, title = {LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR}, author = {Said Taghadouini and Adrien Cavaill\`{e}s and Baptiste Aubertin}, year = {2026}, howpublished = {\url{[https://arxiv.org/abs/2601.14251](https://arxiv.org/abs/2601.14251)}}} @article{cheng2023optimize, title={Optimize Weight Rounding via Signed Gradient Descent for Large Language Model Quantization}, author={Cheng, Wenhua and Hai, Wei and Shen, Haihao and others}, journal={arXiv preprint arXiv:2309.05516}, year={2023}}

LightOnOCR-3-4B-W4A16-AutoRound-GPTQ

README

Quantization Details

Sibling Repositories & Formats

Operating Modes

Quickstart

1. Python Inference via Transformers

2. High-Throughput Serving with vLLM

Start Server

Query Server

Best Practices for Document Preprocessing

🚀 Deploy on RunPod

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

Citation & Acknowledgements

Explore FriendliAI today

README

Quantization Details

Sibling Repositories & Formats

Operating Modes

Quickstart

1. Python Inference via Transformers

2. High-Throughput Serving with vLLM

Start Server

Query Server

Best Practices for Document Preprocessing

🚀 Deploy on RunPod

PyTorch 2.14

PyTorch 2.13

PyTorch 2.12

Citation & Acknowledgements