Benchmark
Measured on an NVIDIA H200 MIG 3g.71gb slice, 17 pages
(17 with ground truth), 1 warm-up + 3 timed runs per page.
Table with columns: bf16 (original), NF4 (this repo) | bf16 (original) | NF4 (this repo) |
|---|
| Peak GPU memory | 7.39 GB | 3.17 GB (-57%) |
| Time per page | 6.73 s | 7.86 s (+17%) |
| Tokens / second | 38.0 | 32.5 |
| Mean CER | 0.069% | 0.068% |
| Mean WER | 0.247% | 0.234% |
The test set is small, so CER differences of a few characters are not significant.
Requirements
- NVIDIA GPU with CUDA (bitsandbytes 4-bit does not run on CPU or Apple Silicon)
torch==2.13.0 (newer torch ships Triton 3.9, which breaks the model's attention kernels)
transformers==4.52.1, bitsandbytes, torchvision, einops, addict, easydict
Usage
import torchfrom transformers import AutoModel, AutoTokenizer repo = "aungthuhein-dev/Unlimited-OCR-2-bnb-nf4"tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)model = AutoModel.from_pretrained( repo, trust_remote_code=True, device_map="auto", torch_dtype=torch.bfloat16,).eval() text = model.infer(tokenizer, image_file="page.png", eval_mode=True)print(text)
No quantization_config is needed: the settings are stored in config.json.
torch_dtype=torch.bfloat16 is required: without it the non-quantized layers load in float16 and
inference fails with Input type (c10::BFloat16) and bias type (c10::Half) should be the same.
License and attribution
The original model is released by Baidu under the MIT license; this quantized copy keeps that
license. This is an independent quantization and is not affiliated with or endorsed by Baidu.