Vishva007

LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ

README

License: apache-2.0

Quantization Details

Table with columns: Parameter, Configuration, Details
Parameter	Configuration	Details
Quantization Scheme	`W4A16`	Weights INT4, Activations BF16
Group Size	`32`	Enhanced accuracy per weight block
Symmetry	`Symmetric` (`sym=True`)	Standard high-throughput configuration
Iterations	`1200`	Production-grade fine-grained optimization
Calibration Samples	`512` samples (`seqlen=4096`)	Broad domain calibration coverage
Vision Preservation	`quant_nontext_module=False`	Vision tower & multimodal projector kept in BF16 to prevent OCR/grounding degradation

Sibling Repositories & Formats

Choose the backend format suited to your inference engine:

AutoRound Native / Transformers: Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound
AutoGPTQ / Marlin: Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ
vLLM / Compressed Tensors: Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-LLM-Compressor

Operating Modes

LightOnOCR-3 operates in two primary modes:

Plain Transcription Mode (prompt = ""): Transcribes text directly to Markdown layout without bounding boxes.
Grounding Mode (prompt = "grounding"): Outputs normalized bounding boxes [0, 1000] and semantic labels (text, title, table, image, chart, etc.) for each document component.

Quickstart

1. Python Inference via Transformers

bash

pip install --upgrade "transformers>=5.5.4" auto-round pillow torchvision

python

import torchfrom transformers import AutoProcessor, AutoModelForImageTextToText # Swap repo_id with your target variant:model_id = "Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ" processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(    model_id,    torch_dtype=torch.bfloat16,    device_map="auto") # Set image URL or local pathimage_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" # Use "grounding" or "" (for transcription only)prompt = "grounding" content = [{"type": "image", "url": image_url}]if prompt:    content.append({"type": "text", "text": prompt}) conversation = [{"role": "user", "content": content}] inputs = processor.apply_chat_template(    conversation,    add_generation_prompt=True,    tokenize=True,    return_dict=True,    return_tensors="pt",    enable_thinking=False,  # LightOnOCR is optimized with thinking disabled).to(model.device) with torch.inference_mode():    output_ids = model.generate(**inputs, max_new_tokens=2048, do_sample=False) generated_ids = output_ids[0, inputs["input_ids"].shape[1]:]print(processor.decode(generated_ids, skip_special_tokens=True))

2. High-Throughput Serving with vLLM

(Recommended using Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-LLM-Compressor or Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ)

Start Server

bash

vllm serve Vishva007/Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ \  --trust-remote-code \  --dtype bfloat16 \  --max-model-len 8192 \  --default-chat-template-kwargs '{"enable_thinking": false}'

Query Server

python

import requests url = "http://localhost:8000/v1/chat/completions"image_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" payload = {    "model": "Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ",    "messages": [        {            "role": "user",            "content": [                {"type": "image_url", "image_url": {"url": image_url}},                {"type": "text", "text": "grounding"}            ]        }    ],    "max_tokens": 2048,    "temperature": 0.0} response = requests.post(url, json=payload)print(response.json()["choices"][0]["message"]["content"])

Best Practices for Document Preprocessing

Resolution: Render scanned documents and PDFs at 400 DPI using a target longest dimension of 2048px.
Aspect Ratio: Maintain the native aspect ratio when resizing; do not stretch images, as bounding box coordinate accuracy depends on geometric integrity.
Thinking Tokens: Ensure enable_thinking=False is preserved in chat templates during generation.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5-$ 500 in free credits when you add your first $10.

PyTorch 2.14

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`

PyTorch 2.13

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`

PyTorch 2.12

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`

Citation & Acknowledgements

Quantized using Intel AutoRound. Base model created and released by LightOn AI.

bibtex

@misc{lightonocr2_2026,  title        = {LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR},  author       = {Said Taghadouini and Adrien Cavaill\`{e}s and Baptiste Aubertin},  year         = {2026},  howpublished = {\url{[https://arxiv.org/abs/2601.14251](https://arxiv.org/abs/2601.14251)}}} @article{cheng2023optimize,  title={Optimize Weight Rounding via Signed Gradient Descent for Large Language Model Quantization},  author={Cheng, Wenhua and Hai, Wei and Shen, Haihao and others},  journal={arXiv preprint arXiv:2309.05516},  year={2023}}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

Vishva007

Model Tree

Base

lightonai/LightOnOCR-3-0.8B

Quantized

this model

Input Modalities

TextImageVideo

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

Quantization Details

Table with columns: Parameter, Configuration, Details
Parameter	Configuration	Details
Quantization Scheme	`W4A16`	Weights INT4, Activations BF16
Group Size	`32`	Enhanced accuracy per weight block
Symmetry	`Symmetric` (`sym=True`)	Standard high-throughput configuration
Iterations	`1200`	Production-grade fine-grained optimization
Calibration Samples	`512` samples (`seqlen=4096`)	Broad domain calibration coverage
Vision Preservation	`quant_nontext_module=False`	Vision tower & multimodal projector kept in BF16 to prevent OCR/grounding degradation

Sibling Repositories & Formats

Choose the backend format suited to your inference engine:

AutoRound Native / Transformers: Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound
AutoGPTQ / Marlin: Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ
vLLM / Compressed Tensors: Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-LLM-Compressor

Operating Modes

LightOnOCR-3 operates in two primary modes:

Plain Transcription Mode (prompt = ""): Transcribes text directly to Markdown layout without bounding boxes.
Grounding Mode (prompt = "grounding"): Outputs normalized bounding boxes [0, 1000] and semantic labels (text, title, table, image, chart, etc.) for each document component.

Quickstart

1. Python Inference via Transformers

bash

pip install --upgrade "transformers>=5.5.4" auto-round pillow torchvision

pip install --upgrade "transformers>=5.5.4" auto-round pillow torchvision

python

import torchfrom transformers import AutoProcessor, AutoModelForImageTextToText # Swap repo_id with your target variant:model_id = "Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ" processor = AutoProcessor.from_pretrained(model_id)model = AutoModelForImageTextToText.from_pretrained(    model_id,    torch_dtype=torch.bfloat16,    device_map="auto") # Set image URL or local pathimage_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" # Use "grounding" or "" (for transcription only)prompt = "grounding" content = [{"type": "image", "url": image_url}]if prompt:    content.append({"type": "text", "text": prompt}) conversation = [{"role": "user", "content": content}] inputs = processor.apply_chat_template(    conversation,    add_generation_prompt=True,    tokenize=True,    return_dict=True,    return_tensors="pt",    enable_thinking=False,  # LightOnOCR is optimized with thinking disabled).to(model.device) with torch.inference_mode():    output_ids = model.generate(**inputs, max_new_tokens=2048, do_sample=False) generated_ids = output_ids[0, inputs["input_ids"].shape[1]:]print(processor.decode(generated_ids, skip_special_tokens=True))

2. High-Throughput Serving with vLLM

(Recommended using Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-LLM-Compressor or Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ)

Start Server

bash

vllm serve Vishva007/Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ \  --trust-remote-code \  --dtype bfloat16 \  --max-model-len 8192 \  --default-chat-template-kwargs '{"enable_thinking": false}'

vllm serve Vishva007/Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ \  --trust-remote-code \  --dtype bfloat16 \  --max-model-len 8192 \  --default-chat-template-kwargs '{"enable_thinking": false}'

Query Server

python

import requests url = "http://localhost:8000/v1/chat/completions"image_url = "[https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg](https://huggingface.co/datasets/hf-internal-testing/fixtures_ocr/resolve/main/SROIE-receipt.jpeg)" payload = {    "model": "Vishva007/LightOnOCR-3-0.8B-W4A16-AutoRound-GPTQ",    "messages": [        {            "role": "user",            "content": [                {"type": "image_url", "image_url": {"url": image_url}},                {"type": "text", "text": "grounding"}            ]        }    ],    "max_tokens": 2048,    "temperature": 0.0} response = requests.post(url, json=payload)print(response.json()["choices"][0]["message"]["content"])

Best Practices for Document Preprocessing

Resolution: Render scanned documents and PDFs at 400 DPI using a target longest dimension of 2048px.
Aspect Ratio: Maintain the native aspect ratio when resizing; do not stretch images, as bounding box coordinate accuracy depends on geometric integrity.
Thinking Tokens: Ensure enable_thinking=False is preserved in chat templates during generation.

🚀 Deploy on RunPod

One-click launch environments pre-configured with PyTorch, CUDA, and dependencies for fine-tuning or quantization.

🎁 Need GPU compute? Sign up via RunPod and get $5-$ 500 in free credits when you add your first $10.

PyTorch 2.14

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.14 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.14-runpod`	`d7lxsa4w9m`
PyTorch 2.14 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.14-runpod`

PyTorch 2.13

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.13 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.13-runpod`	`gmlupxnxfk`
PyTorch 2.13 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.13-runpod`

PyTorch 2.12

Table with columns: Template, CUDA Version, Docker Image, Template ID, Deploy
Template	CUDA Version	Docker Image	Template ID	Deploy
PyTorch 2.12 (CUDA 12.6)	12.6	`vishva123/cuda-12.6-pytorch-2.12-runpod`	`ctmz86zmf0`
PyTorch 2.12 (CUDA 13.0)	13.0	`vishva123/cuda-13.0-pytorch-2.12-runpod`

Citation & Acknowledgements

Quantized using Intel AutoRound. Base model created and released by LightOn AI.

bibtex

@misc{lightonocr2_2026,  title        = {LightOnOCR: A 1B End-to-End Multilingual Vision-Language Model for State-of-the-Art OCR},  author       = {Said Taghadouini and Adrien Cavaill\`{e}s and Baptiste Aubertin},  year         = {2026},  howpublished = {\url{[https://arxiv.org/abs/2601.14251](https://arxiv.org/abs/2601.14251)}}} @article{cheng2023optimize,  title={Optimize Weight Rounding via Signed Gradient Descent for Large Language Model Quantization},  author={Cheng, Wenhua and Hai, Wei and Shen, Haihao and others},  journal={arXiv preprint arXiv:2309.05516},  year={2023}}