tevfikk

granite-docling-258M-bitnet

README

License: apache-2.0

📊 Benchmark Highlights

Table with columns: Metric, Original FP16 / BF16, BitNet b1.58 (This Model), Improvement / Difference
Metric	Original FP16 / BF16	BitNet b1.58 (This Model)	Improvement / Difference
Model Size (Language Decoder)	~330 MB	103.0 MB (Packed Ternary)	3.2× Decoder Compression
C99 Standalone Container (`.eifm`)	728 MB	165.0 MB (Dense EIFM)	4.4× Edge Compression
Weight Precision	16-bit Float	1.58-bit Ternary (-1, 0, +1)	10.1× Weight Bit Density
Logit Cosine Similarity	1.0000	0.8912 (89.12%)	Near theoretical limit for 100k vocab
Feature Cosine Similarity (FeatSim)	1.0000	0.7776 (77.76%)	Latent semantic fidelity
Top-1 Token Agreement	`<doctag>` (100327)	`<doctag>` (100327)	100% Exact Match on Layouts
C99 Inference Throughput (AVX2)	~32 tok/s	123.6 - 156.6 tok/s	~3.9× - 4.8× Speedup
Latency per Token (C99)	31.2 ms	6.38 - 8.09 ms	Ultra-responsive streaming

🏗️ Model Architecture

Base Model: ibm-granite/granite-docling-258M
Vision Backbone: SigLIP2 Vision Encoder (frozen BF16 ViT backbone, preserving high-resolution document OCR and layout token embeddings)
Multimodal Connector: Pixel Shuffle Projector (linear dimension connector, 4× spatial reduction)
Language Decoder: Granite 165M Decoder (GQA 8:2, 12 layers, hidden size 768) converted completely to BitLinear b1.58 layers
Ternary Packing: 5 ternary weights $\in {- 1, 0, + 1}$ packed per byte ( $3$ )

🔬 Training & Distillation Methodology

1. Straight-Through Estimator (STE) Quantization

Weights are ternary-quantized dynamically with per-tensor mean absolute scale: $W ternary = Round (Clip (γ W , - 1, 1)), γ$ Activations are scaled to 8-bit integers per token using absolute maximum scaling:

2. Multi-Objective Distillation Loss

Supervised Cross-Entropy ( $L CE $ ): Next-token prediction on valid assistant DocTags (converged to $\sim 0.39 - 0.49$ ).
KL-Divergence Distillation ( $L KD $ ): Full-vocabulary logit matching at temperature $T =$ (converged to ).

📚 Training Datasets

Trained directly on official IBM document extraction and layout datasets:

HuggingFaceM4/DoclingMatix (50%): Real multi-page document images, PDFs, invoices, forms, structured document QA, and DocTags layout annotations.
ds4sd/SynthCodeNet (20%): Synthesized code blocks with pixel bounding-box DocTags (<code><loc_...>).
ds4sd/SynthFormulaNet (15%): Dense LaTeX mathematical formulas with layout coordinates (<formula><loc_...>).
ds4sd/SynthChartNet (15%): Charts, infographics, and table structure extractions with cell coordinates.

📈 Training History & Token Statistics

The model underwent four cumulative training stages with an effective batch size of 64 document images per step.

Table with columns: Phase, Description, Steps, Effective Batch Size, Document Samples, Multimodal Context Tokens, Supervised DocTag Tokens
Phase	Description	Steps	Effective Batch Size	Document Samples	Multimodal Context Tokens	Supervised DocTag Tokens
Phase 1	Pre-training & VQA Distillation	4,672	64 (32 × 2)	299,008	~285.1M	~27.8M
Phase 2	Targeted QAT DocTags Tuning	1,000	64 (4 × 16)	64,000

Average multimodal sequence length: ~953 tokens/sample (vision patches + prompt + DocTags).
Directly optimized DocTags output: ~93 tokens/sample.
Total Training Tokens Processed: ~506.3 Million multimodal tokens across all cumulative phases.
Convergence State: The model reached diminishing returns with sustained logit cosine similarity of 0.8912, saturating the parameter capacity of 1.58-bit ternary representations.

🖥️ Hardware & Training Infrastructure

The training pipeline was executed on the following infrastructure:

Hardware: NVIDIA GB10 (12 GB VRAM)
Active VRAM Usage: ~3.0 GB active / ~11.0 GB reserved (via selective token masking & gradient checkpointing)
Optimizer: 8-bit PagedAdamW (bitsandbytes) with weight decay $1 \times 1 0 - 4$
Learning Rate Schedule: Warm restart cosine annealing ( $LR m a x =$ , )

🛠️ Usage via Hugging Face (`transformers`)

python

import torchfrom transformers import AutoProcessor, Idefics3ForConditionalGenerationfrom PIL import Image device = "cuda" if torch.cuda.is_available() else "cpu"model_id = "tevfikk/granite-docling-258M-bitnet" processor = AutoProcessor.from_pretrained(model_id)model = Idefics3ForConditionalGeneration.from_pretrained(    model_id,    torch_dtype=torch.bfloat16,    device_map=device) image = Image.open("sample_document.png").convert("RGB")prompt = (    "<|start_of_role|>user<|end_of_role|><image>Convert this page to docling.<|end_of_text|>\n"    "<|start_of_role|>assistant<|end_of_role|>") inputs = processor(text=prompt, images=[image], return_tensors="pt").to(device)with torch.no_grad():    generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False) output = processor.batch_decode(    generated_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=False)[0]print(output)# Output: <doctag><section_header_level_1>INVOICE #9812</section_header_level_1>...

⚡ Standalone C99 Edge Binary (`eif-runtime`)

For microcontrollers, Raspberry Pi, and edge accelerators without Python or CUDA:

bash

# 1. Clone & build the zero-dependency C99 runtimegit clone https://github.com/tevfik/eif-runtime.gitcd eif-runtime && mkdir build && cd build && cmake .. && make -j # 2. Run at 156+ tokens/sec using AVX2/NEON multiplication-free kernel./bin/eif_infer --model granite_docling_bitnet_dense.eifm --image sample_document.png

📜 Citation & Credits

Base Model: IBM Granite Docling 258M
BitNet Architecture: The Era of 1-bit LLMs (Microsoft Research)
Distillation Framework & Runtime: Edge-Intelligence Project & eif-runtime

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Model Details

Model Provider

tevfikk

Model Tree

Base

ibm-granite/granite-docling-258M

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated Endpoints

Explore FriendliAI today

Get started Talk to an engineer

README

License: apache-2.0

📊 Benchmark Highlights

Table with columns: Metric, Original FP16 / BF16, BitNet b1.58 (This Model), Improvement / Difference
Metric	Original FP16 / BF16	BitNet b1.58 (This Model)	Improvement / Difference
Model Size (Language Decoder)	~330 MB	103.0 MB (Packed Ternary)	3.2× Decoder Compression
C99 Standalone Container (`.eifm`)	728 MB	165.0 MB (Dense EIFM)	4.4× Edge Compression
Weight Precision	16-bit Float	1.58-bit Ternary (-1, 0, +1)	10.1× Weight Bit Density
Logit Cosine Similarity	1.0000	0.8912 (89.12%)	Near theoretical limit for 100k vocab
Feature Cosine Similarity (FeatSim)	1.0000	0.7776 (77.76%)	Latent semantic fidelity
Top-1 Token Agreement	`<doctag>` (100327)	`<doctag>` (100327)	100% Exact Match on Layouts
C99 Inference Throughput (AVX2)	~32 tok/s	123.6 - 156.6 tok/s	~3.9× - 4.8× Speedup
Latency per Token (C99)	31.2 ms	6.38 - 8.09 ms	Ultra-responsive streaming

🏗️ Model Architecture

Base Model: ibm-granite/granite-docling-258M
Vision Backbone: SigLIP2 Vision Encoder (frozen BF16 ViT backbone, preserving high-resolution document OCR and layout token embeddings)
Multimodal Connector: Pixel Shuffle Projector (linear dimension connector, 4× spatial reduction)
Language Decoder: Granite 165M Decoder (GQA 8:2, 12 layers, hidden size 768) converted completely to BitLinear b1.58 layers
Ternary Packing: 5 ternary weights $\in {- 1, 0, + 1}$ packed per byte ( $3$ )

🔬 Training & Distillation Methodology

1. Straight-Through Estimator (STE) Quantization

2. Multi-Objective Distillation Loss

Supervised Cross-Entropy ( $L CE $ ): Next-token prediction on valid assistant DocTags (converged to $\sim 0.39 - 0.49$ ).
KL-Divergence Distillation ( $L KD $ ): Full-vocabulary logit matching at temperature $T =$ (converged to ).

📚 Training Datasets

Trained directly on official IBM document extraction and layout datasets:

HuggingFaceM4/DoclingMatix (50%): Real multi-page document images, PDFs, invoices, forms, structured document QA, and DocTags layout annotations.
ds4sd/SynthCodeNet (20%): Synthesized code blocks with pixel bounding-box DocTags (<code><loc_...>).
ds4sd/SynthFormulaNet (15%): Dense LaTeX mathematical formulas with layout coordinates (<formula><loc_...>).
ds4sd/SynthChartNet (15%): Charts, infographics, and table structure extractions with cell coordinates.

📈 Training History & Token Statistics

The model underwent four cumulative training stages with an effective batch size of 64 document images per step.

Table with columns: Phase, Description, Steps, Effective Batch Size, Document Samples, Multimodal Context Tokens, Supervised DocTag Tokens
Phase	Description	Steps	Effective Batch Size	Document Samples	Multimodal Context Tokens	Supervised DocTag Tokens
Phase 1	Pre-training & VQA Distillation	4,672	64 (32 × 2)	299,008	~285.1M	~27.8M
Phase 2	Targeted QAT DocTags Tuning	1,000	64 (4 × 16)	64,000

Average multimodal sequence length: ~953 tokens/sample (vision patches + prompt + DocTags).
Directly optimized DocTags output: ~93 tokens/sample.
Total Training Tokens Processed: ~506.3 Million multimodal tokens across all cumulative phases.
Convergence State: The model reached diminishing returns with sustained logit cosine similarity of 0.8912, saturating the parameter capacity of 1.58-bit ternary representations.

🖥️ Hardware & Training Infrastructure

The training pipeline was executed on the following infrastructure:

Hardware: NVIDIA GB10 (12 GB VRAM)
Active VRAM Usage: ~3.0 GB active / ~11.0 GB reserved (via selective token masking & gradient checkpointing)
Optimizer: 8-bit PagedAdamW (bitsandbytes) with weight decay $1 \times 1 0 - 4$
Learning Rate Schedule: Warm restart cosine annealing ( $LR m a x =$ , )

🛠️ Usage via Hugging Face (`transformers`)

python

import torchfrom transformers import AutoProcessor, Idefics3ForConditionalGenerationfrom PIL import Image device = "cuda" if torch.cuda.is_available() else "cpu"model_id = "tevfikk/granite-docling-258M-bitnet" processor = AutoProcessor.from_pretrained(model_id)model = Idefics3ForConditionalGeneration.from_pretrained(    model_id,    torch_dtype=torch.bfloat16,    device_map=device) image = Image.open("sample_document.png").convert("RGB")prompt = (    "<|start_of_role|>user<|end_of_role|><image>Convert this page to docling.<|end_of_text|>\n"    "<|start_of_role|>assistant<|end_of_role|>") inputs = processor(text=prompt, images=[image], return_tensors="pt").to(device)with torch.no_grad():    generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False) output = processor.batch_decode(    generated_ids[:, inputs.input_ids.shape[1]:], skip_special_tokens=False)[0]print(output)# Output: <doctag><section_header_level_1>INVOICE #9812</section_header_level_1>...

⚡ Standalone C99 Edge Binary (`eif-runtime`)

For microcontrollers, Raspberry Pi, and edge accelerators without Python or CUDA:

bash

# 1. Clone & build the zero-dependency C99 runtimegit clone https://github.com/tevfik/eif-runtime.gitcd eif-runtime && mkdir build && cd build && cmake .. && make -j # 2. Run at 156+ tokens/sec using AVX2/NEON multiplication-free kernel./bin/eif_infer --model granite_docling_bitnet_dense.eifm --image sample_document.png

📜 Citation & Credits

Base Model: IBM Granite Docling 258M
BitNet Architecture: The Era of 1-bit LLMs (Microsoft Research)
Distillation Framework & Runtime: Edge-Intelligence Project & eif-runtime

granite-docling-258M-bitnet

README

📊 Benchmark Highlights

🏗️ Model Architecture

🔬 Training & Distillation Methodology

1. Straight-Through Estimator (STE) Quantization

2. Multi-Objective Distillation Loss

📚 Training Datasets

📈 Training History & Token Statistics

🖥️ Hardware & Training Infrastructure

🛠️ Usage via Hugging Face (transformers)

⚡ Standalone C99 Edge Binary (eif-runtime)

📜 Citation & Credits

Explore FriendliAI today

README

📊 Benchmark Highlights

🏗️ Model Architecture

🔬 Training & Distillation Methodology

1. Straight-Through Estimator (STE) Quantization

2. Multi-Objective Distillation Loss

📚 Training Datasets

📈 Training History & Token Statistics

🖥️ Hardware & Training Infrastructure

🛠️ Usage via Hugging Face (transformers)

⚡ Standalone C99 Edge Binary (eif-runtime)

📜 Citation & Credits

🛠️ Usage via Hugging Face (`transformers`)

⚡ Standalone C99 Edge Binary (`eif-runtime`)

🛠️ Usage via Hugging Face (`transformers`)

⚡ Standalone C99 Edge Binary (`eif-runtime`)