What is quantized
Scheme: FP8_DYNAMIC via llm-compressor —
per-channel FP8 weights, dynamic per-token FP8 activations, data-free. Only the DeepSeek-V2 MoE
text-decoder linears are quantized (2196 modules: 2112 routed-expert, 33 shared-expert/dense-MLP,
48 attention, 3 dense-layer). Everything the vision path depends on stays BF16:
- SAM-ViT-B + CLIP-L DeepEncoder vision tower and projector
embed_tokens / lm_head
- MoE router gates and all norms
Validation
Greedy OCR on the fixture set (vLLM 0.26.0, SM120) is character-identical to the BF16
baseline — CER 0.0 on every fixture, grounding-box coordinates included. The same parity holds
under transformers + compressed-tensors.
Table with columns: BF16, FP8-Dynamic (this repo) | BF16 | FP8-Dynamic (this repo) |
|---|
| Mean CER vs BF16 | — | 0.0000 |
| invoice / memo / table CER | — | 0.0 / 0.0 / 0.0 |
| Decode throughput (tok/s, greedy) | 45.3 | 39.4 |
| Checkpoint size | 6.7 GB | 3.9 GB |
Raw per-fixture transcripts ship under evidence/, produced by the
validation harness.
Note: the shipped quantization_config.ignore uses prefix-agnostic regex patterns — required for
vLLM to correctly skip the vision tower for quantization compatibility. Do not replace it with expanded
module names.
Usage
import torchfrom transformers import AutoModel, AutoTokenizer repo = "shadowrock-io/Unlimited-OCR-Community-FP8-Dynamic"tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)model = AutoModel.from_pretrained(repo, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda").eval()text = model.infer(tok, prompt="<image>\n<|grounding|>OCR this image.", image_file="document.png", output_path="./out", base_size=1024, image_size=1024, crop_mode=False, eval_mode=True)
About
Quantized by Matt Busi at ShadowRock. Raw evaluation outputs ship under
evidence/.