Agreement with the original on real documents
Table with columns: Model, Size on disk, Pages within 1% of the original's text, Character difference vs the original: median / mean| Model | Size on disk | Pages within 1% of the original's text | Character difference vs the original: median / mean |
|---|
| W4A16, symmetric int4, group 32 (this repo) | 3.06 GB | 23/30 | 0.13% / 1.06% |
| W4A16, symmetric int4, group 128 (first upload) | 2.96 GB | 18/30 | 0.11% / 3.42% |
Measured on 30 real pages from OmniDocBench (2 per category: textbooks, newspapers, slides, reports, notes, exams,
books, magazines, papers and more; seed 0), one page per request, against the original model in BF16 with the
vendor's code (transformers 4.52.1). The original gives the same text on every run (30/30 pages identical across
two runs), so these differences come from the 4-bit weights. Character difference = edit distance of the
recognized text after removing layout headers (block type and box), divided by the original's length.
The repo carries the vendor's remote code and the original config.json (plus the quantization block):
import torchfrom compressed_tensors.linear.compressed_linear import CompressedLinearfrom transformers import AutoModel, AutoTokenizer name = "AtomicChat/Unlimited-OCR-2-W4A16"tok = AutoTokenizer.from_pretrained(name, trust_remote_code=True)model = AutoModel.from_pretrained(name, trust_remote_code=True, torch_dtype=torch.bfloat16).eval().cuda() # unpack the 4-bit layers into plain BF16 Linear once (~3 s)for parent in list(model.modules()): for child_name, child in list(parent.named_children()): if isinstance(child, CompressedLinear): lin = torch.nn.Linear(child.in_features, child.out_features, bias=child.bias is not None, device="cuda", dtype=torch.bfloat16) lin.weight.data = child.compressor.decompress_module(child).to(torch.bfloat16) setattr(parent, child_name, lin) text = model.infer(tok, image_file="page.png", output_path="out", save_results=True, max_new_tokens=8192)
- Do not pass
CompressedTensorsConfig(run_compressed=False) with transformers 4.52.1. On this MoE (about 2,100
quantized expert layers) the loader's key check becomes quadratic and stalls for hours. The unpack loop above
gives the same weights in about 10 s.
- In transformers the gain is the download and disk size, not GPU memory.
- Cap
max_new_tokens: the vendor default allows up to 131k new tokens per call.
- Do not re-save the loaded model with
save_pretrained. The vendor code rewrites sparse_attention_config at
load time (dash_stage0, dash_topk_blocks), and a re-saved config makes the model emit garbage.
Serving engines. The vendor's own SGLang build serves Unlimited-OCR-2 in BF16 only: its MoE path flattens
the experts' .weight tensors and cannot load quantized experts. vLLM has no v2 support yet. For local use there
are GGUF files: AtomicChat/Unlimited-OCR-2-GGUF.
License
MIT, inherited from baidu/Unlimited-OCR-2. Built by AtomicChat.