What is quantized
Scheme: NVFP4 via llm-compressor GPTQ
(offloaded hessians, 32 calibration sequences, max length 2048). Only the DeepSeek-V2 MoE
text-decoder linears are quantized (2196 modules). Kept in BF16:
- SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
embed_tokens / lm_head
- MoE router gates and all norms
Three of 768 routed-expert instances were never activated by the calibration corpus; their
weights are quantized data-free from the original BF16 checkpoint and their activation scales
derived from sibling-expert statistics (see source-repo issue #6 and
quantization/repair_dead_experts.py). All other experts are GPTQ-calibrated.
Validation
Greedy OCR on the fixture set (vLLM 0.26.0, SM120, Marlin NVFP4 MoE backend), CER vs the BF16
baseline after stripping grounding tags:
Table with columns: BF16, Data-free community NVFP4, This repo (calibrated) | BF16 | Data-free community NVFP4 | This repo (calibrated) |
|---|
| Mean CER vs BF16 | — | 4.27* | 0.0068 |
| invoice / memo / table CER | — | unstable* | 0.0 / 0.0 / 0.02 |
| Decode throughput (tok/s, greedy) | 45.3 | 64.2* | 40.8 |
| Checkpoint size | 6.7 GB | 2.8 GB | 2.8 GB |
* Prior community data-free build measured on the same harness: runaway repetition to the token
cap on two fixtures and immediate EOS on the third — its higher tok/s reflects degenerate
generation, not usable speed.
Two of three fixtures are character-identical to BF16 including box coordinates; the third
differs by a single short span. See evidence/ for raw
per-fixture transcripts, CER vs BF16, decode throughput, and peak VRAM from the
validation harness. Calibration corpus and
provenance ship with the repo (QUANT_PROVENANCE.json).
Usage
import torchfrom transformers import AutoModel, AutoTokenizer repo = "shadowrock-io/Unlimited-OCR-Community-NVFP4"tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)model = AutoModel.from_pretrained(repo, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda").eval()text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.", image_file="document.png", output_path="./out", base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)
Note: transformers loads of NVFP4 checkpoints may require TORCH_COMPILE_DISABLE=1.
About
Quantized by Matt Busi at ShadowRock. Raw evaluation outputs ship under
evidence/.