Files
Table with columns: File, Size, Use| File | Size | Use |
|---|
model.safetensors | 728 MB | Training checkpoint (step 5500): latent FP32 master weights of the decoder + SubLN norms + vision tower + connector. Load it with modeling_granite_docling_bitnet.py. |
modeling_granite_docling_bitnet.py | 7 KB | Standalone loader (torch + transformers only) that reproduces the trained ternary forward pass. |
granite_docling_bitnet_dense.eifm | 264 MB | Model for bitllm.c: ternary decoder (2-bit packed), INT8 embeddings, INT8 per-row ViT projections, FP32 connector/norms. |
tokenizer.bin | 1.5 MB | Byte-level BPE tokenizer for bitllm.c. |
| tokenizer / processor / config files | | Standard Hugging Face files. |
Results
The benchmark numbers below were measured on this release (step 5500, 2026-10-19, the published files through the bundled PyTorch loader) against the original BF16 model on the same inputs. The real-document table further down was measured on the previous checkpoint (step 3000) and was not repeated for step 5500.
Scripts: models/vlm/granite_docling/eval_docling.py, eval_plugin_docs.py in the
edge-intelligence repository.
Benchmarks (greedy decoding)
FinTabNet and PubTables-1M samples come from the datasets' test splits (training used the train splits); DocLayNet pages come from its
test split and DocLayNet is not among the training sources listed below. Charts are random samples of the SynthChartNet train
split, which is also a training source, so that row is not guaranteed unseen. n = 30 per row, 10 pages. The FinTabNet set is the same one used for the step 3000 numbers (F1 0.816, structure 0.866); PubTables, charts and pages are fresh seeded samples (fetch_eval_data.py), so their step 3000 values (0.739 / 0.890 / 0.347) are not exactly comparable. The step 1126 column is from older samples.
Table with columns: Task, Metric, BF16 teacher, This model, Earlier version of this model (step 1126)| Task | Metric | BF16 teacher | This model | Earlier version of this model (step 1126) |
|---|
FinTabNet tables (Convert table to OTSL.) | cell-text F1 | 0.943 | 0.801 | 0.376 |
| FinTabNet tables | structure similarity | 0.987 | 0.800 | 0.495 |
| FinTabNet tables | exact structure match | 0.867 | 0.200 | 0.000 |
| PubTables-1M tables | cell-text F1 | 0.953 | 0.769 | 0.14 |
| PubTables-1M tables | structure similarity | 0.963 | 0.755 | – |
| Charts (SynthChartNet) | DocTags similarity | 0.938 | 0.919 | 0.87 |
| Charts | number F1 | 0.701 | 0.632 | – |
DocLayNet pages (Convert this page to docling.) | word recall | 0.883 | 0.461 | 0.11 |
Table structure is the weak point: cell texts are mostly right, but rows/columns are often misaligned (exact structure match 20% on
FinTabNet vs 87% for the teacher).
Real documents through gleann-plugin-docs (bitllm.c, CPU; step 3000 checkpoint)
The plugin's convert was run on the documents in its testdata; the markdown is scored against the document's own text layer
(word recall = share of the reference words that appear in the output). scanned_report.pdf has no text layer; its reference is the
digital twin (benchmark_report.pdf, page 1). The other two cases force the model onto digital pages.
Table with columns: Document, Earlier version (step 1126), Step 3000, INT8 ViT (shipped format), Step 3000, INT4 ViT| Document | Earlier version (step 1126) | Step 3000, INT8 ViT (shipped format) | Step 3000, INT4 ViT |
|---|
| Scanned report page (1 page, English) | 0.095 | 0.968 | 0.847 |
| Digital report forced through the VLM (2 pages) | 0.350 | 0.978 | 0.957 |
| Dense bilingual (Turkish/English) academic paper, 2 pages | 0.018 | 0.093 | 0.031 |
| same pages, BF16 teacher (PyTorch) | | 0.416 | |
The shipped .eifm uses INT8 vision weights. INT4 (g128 + Hadamard rotation) is about 40 MB smaller but visibly worse (garbled words,
repetition loops on dense text) and is not shipped; docling_to_eif.py --vision-weights int4-g128-had produces it.
Size and speed (CPU, 20-core ARM, 10 threads, INT8 ViT, bitllm.c)
Table with columns: BF16 original, This model | BF16 original | This model |
|---|
| Whole model file | 728 MB (safetensors) | 264 MB (.eifm) |
| Language decoder projections (106 M weights) | 212 MB | 26.5 MB ternary (2-bit packed) |
| Vision tower + connector | 187 MB | 114 MB |
| Peak memory (one page, 50 generated tokens) | | ~0.4 GB |
| One page (13 tiles): vision encoding / prompt prefill / decoding | | ~7-9 s / 2.6 s / ~62 tokens/s |
The vision encoder is the largest single cost of a page. With bitllm.c commit cef6d8a or newer the INT8 ViT runs real INT8 matrix products (Armv8.6 i8mm), about 3x faster than the FP32 path (ViT ~4.6 s vs 13-15 s for a 17-tile page). Numbers above the table are from an older commit; a whole page now takes roughly 12-16 s on an idle 20-core ARM (ViT 4-5 s, prefill ~3 s, decode ~4 s), and timings vary a lot when other processes share the cores. On x86 or older ARM cores without i8mm the ViT falls back to the slower path.
Limitations
- Turkish is only partly supported. Phase 6 added Turkish synthetic pages: on a Turkish test page the headings are now read (with occasional wrong letters and one invalid UTF-8 byte), but a dense body paragraph still collapses into a repeating garbled loop. The step 3000 numbers on the two-page bilingual paper (teacher 0.42, model 0.05 in PyTorch / 0.09 in the C runtime) were not re-measured for step 5500. Use another OCR for Turkish body text (see "Which path to use" below).
- Dense pages (two-column papers, small print) lose most of the text: page word recall 0.46 vs 0.88 for the teacher. Keep the
generation cap generous and check outputs for early stops.
- Tables: structure is often wrong even when the cell text is right; merged cells are the common failure.
- Specific symbols:
< inside a cell is written as <s> (observed in both the PyTorch and the C runtime, so it is the model).
- Occasional invalid UTF-8 from the decoder on special characters (lone continuation bytes); sanitise outputs.
- No confidence estimate. The model does not signal when it is wrong.
- Training and evaluation used greedy decoding; sampling was not evaluated.
Which path to use
Table with columns: Input, Recommendation| Input | Recommendation |
|---|
| Clean English scan, invoice, single-column page | This model (GLEANN_DOCLING_VLM=1): word recall 0.97 on a scanned report, vs 0.10 without a VLM. |
Table crop (Convert table to OTSL.) | This model: cell text mostly right (F1 0.77-0.80), structure often wrong. |
| Dense two-column papers, Turkish/diacritic text | Not this model (recall 0.09 on a dense bilingual paper). gleann-plugin-docs' ONNX page parser (layout + OCR, ~0.5-1.2 s/page) reached 0.68 recall on the same paper. |
| Digital PDFs | Text layer (pdftotext), no model needed. |
Architecture
Table | |
|---|
| Base | ibm-granite/granite-docling-258M (Idefics3), 257.6 M parameters |
| Vision | SigLIP2 ViT: 12 layers, hidden 768, 16×16 patches, 512×512 tiles (86.4 M); frozen, identical to the original |
| Connector | pixel-shuffle (×4) + linear projection 12 288 → 576 (7.1 M); trained in the student |
| Decoder | Llama-style, 30 layers, hidden 576, 9 heads / 3 KV heads, MLP 1536, vocab 100 352, tied embeddings (164.1 M; 106.2 M ternary projection weights + 57.8 M embedding) |
| Page = 13 tiles | 12 local tiles + 1 global overview = 832 visual tokens |
Quantisation
- Weights (decoder, 210 projections):
W_ternary = clip(round(W / γ), -1, +1) · γ, γ = mean(|W_row|), one scale per output row.
Stored latent in FP32 in model.safetensors, 2-bit packed + FP32 row scales in the .eifm.
- SubLN:
o_proj and down_proj apply a trained RMSNorm to their input (60 norms), as in BitNet b1.58.
- Activations: training simulated 8-bit per-token absmax activations (straight-through estimator). At inference both the
loader and
bitllm.c keep activations in floating point.
- Embeddings / LM head: INT8 per-row in the
.eifm. Vision weights: INT8 per-row in the .eifm (lossless in our tests:
table F1 0.946 vs 0.950); the connector stays FP32.
- Tried, not used: one scale per 128 weights plus a Walsh-Hadamard rotation (Bonsai-style). Switching the trained model to it and
training both variants for 100 steps: KL to the teacher 5.55 vs 0.57, table F1 0.00 vs 0.58, so the per-row quantiser was kept.
bitllm.c supports that layout for models trained with it from scratch.
Training
Quantisation-aware distillation from the frozen BF16 teacher. Loss = 0.25·CE (assistant tokens) + 0.35·KL (T = 2, full vocabulary)
- 0.15·hidden-state cosine + 0.25·logit cosine. Vision features are computed once by the teacher's tower and shared.
- Data mix of the final phase (Phase 6): DoclingMatix 30 %, Turkish Procedural Synthetics 20 % (invoices, receipts, bilateral agreements, 2-column papers, OTSL tables), SynthChartNet 14 %, FinTabNet-OTSL 10 %, PubTables-1M-OTSL 10 %, SynthCodeNet 8 %, SynthFormulaNet 8 %.
- Phases 1-4: Early pipeline exploration.
- Phase 5 (steps 1 127 - 3 000): Corrected distillation alignment, full answers (up to 3 072 tokens), OTSL tables integration.
- Phase 6 (steps 3 001 - 5 500, this release): 2 500 additional distillation steps (~160 k pages) aimed at multilingual alignment and at BPE byte-level corruption on Turkish diacritics (not fully solved, see Limitations) (
ş, ğ, ç, ö, ü, ı, İ). 17 497 zero-replay dynamic procedural Turkish pages were generated and trained on.
- Phase 6 settings: micro-batch 4 × 16 accumulation (64 pages per step), 8-bit PagedAdamW, LR 5e-5 with cosine decay (floor 2.5e-5), ~2.0-2.5 pages/s. Final micro-batch loss ≈ 0.79 (KD 1.43, CE 0.96, CosSim 0.882, FeatSim 0.815).
- Hardware: NVIDIA GB10 (Grace-Blackwell, unified memory), VRAM peak 26.6 GB.
Usage
model.safetensors holds latent weights. Plain Idefics3ForConditionalGeneration.from_pretrained would run them unquantised and
without SubLN, which is a different (and worse) model. Use the bundled loader:
import sysfrom huggingface_hub import snapshot_downloadfrom PIL import Image path = snapshot_download("tevfikk/granite-docling-258M-bitnet")sys.path.insert(0, path)from modeling_granite_docling_bitnet import load_granite_docling_bitnet, convert_page model, processor = load_granite_docling_bitnet(path) # cuda if available, else cpu; bf16page = Image.open("page.png").convert("RGB")print(convert_page(model, processor, page, "Convert this page to docling.", max_new_tokens=1024))# tables: convert_page(model, processor, table_image, "Convert table to OTSL.")
The loader quantises the weights on the target device once; its logits and generations are identical to the training code path
(tested). Needs torch, transformers (tested with 5.x) and safetensors.
C99, no Python: bitllm.c
git clone https://github.com/tevfik/bitllm.c.git && cd bitllm.c # commit b94202e or newercmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build -j./build/bitllm-run --model granite_docling_bitnet_dense.eifm --tokenizer tokenizer.bin \ --image page.png --prompt "Convert this page to docling." --max-tokens 1024 --threads 10# other tasks: --prompt "Convert table to OTSL." (table crops) --prompt "Convert chart to table."
--image implies greedy decoding (-T 0 -r 1.0); sampling or a repetition penalty corrupts DocTags/OTSL.
The DocTags/OTSL text follows the --- Output Streaming --- line on stdout (the banner and statistics go to stdout too; progress
logs go to stderr). PDFs have to be rendered to images first (e.g. pdftoppm -r 150 -png).
Documents end to end: gleann-plugin-docs
gleann-plugin-docs embeds bitllm.c and converts PDF/DOCX/XLSX/PPTX/images into
markdown and a section graph for gleann. Digital PDFs use the text layer; scanned pages use this model:
GLEANN_DOCLING_VLM=1 GLEANN_DOCLING_MODEL=/path/to/granite_docling_bitnet_dense.eifm \ ./bin/gleann-plugin-docs convert scan.pdf
Reproduce
python models/vlm/granite_docling/eval_docling.py --data <eval pickles> --checkpoint <step_3000.pt> # benchmarkspython models/vlm/granite_docling/eval_plugin_docs.py --plugin <bin> --docs <testdata> --model name=<dir> # real documentspython -m eif_tools.convert.llm.docling_to_eif --model-dir <pkg> --output model.eifm --vision-weights int8 --tokenizer-out tokenizer.bin
Citation & credits