isaacmg

qwen3-vl-8b-hebrew-v19a-ckpt

Deploy Dedicated

README

License: apache-2.0

What the adapter contains

Language LoRA (r16, 252 tensors) and — unlike many VLM LoRA recipes — a live vision-tower LoRA (r16, 108 tensors, all nonzero). Qwen3-VL's vision blocks receive no gradients under common gradient-checkpointing setups; this series applies a patch-embed require-grad fix, and every shipped checkpoint is verified post-hoc at the weight level.
The vision→language merger stack is frozen in v1.9a (a controlled merger-training variant was evaluated separately and not adopted).

Training

Warm start: v1.8b step-700 adapter (weights only, fresh optimizer).
Mixture: 30% Genizah manuscript pages (word-level-verified transcriptions), 20% documentary Genizah fragments, 20% synthetic Hebrew script (multi-font, degradation-augmented), 8% page regions, 7% crops, 10% Talmud small-script pages, 5% Talmud gemara crops.
Schedule: 2000-step cosine at 5e-5, bs 1 × grad-accum 8, QLoRA on an A100; run ended at step 1310; step 1300 is canonical (eval loss 0.554).
Resolution contract: images are processed at a 6.5–7.0 MP floor (min_pixels=6_500_000, max_pixels=7_000_000). Fine-tuned VLMs in this series are resolution-brittle: inference must use the same processor policy the model was trained with.

Results (internal benchmarks, Aug 2026)

Scored with an alignment-based offline scorer (failure-mode classification; aligned F1; order-independent n-gram precision; CER conditional on a substantive attempt — never bare mean CER).

Genizah religious-text benchmark — 140 held-out manuscript pages (Talmud/Mishnah/liturgy; multi-column pages included; ground truth from word-level scholarly transcriptions, geometry-reconstructed, rev 1.1; three-layer decontamination against all training corpora):

Table with columns: system, aligned F1 (median), CER*
system	aligned F1 (median)	CER*
v1.9a step-1300	0.816	0.216
Kraken HTR (MiDRASH model) + reading-order reconstruction	0.489	0.375
Kraken HTR, raw output order	0.421	0.519

Frozen documentary benchmark — 131 verified Genizah fragments (Judeo-Arabic-heavy):

Table with columns: step, substantive attempts, aligned F1 (median), CER*
step	substantive attempts	aligned F1 (median)	CER*
1300	98%	0.862	0.196
700	97%	0.854	0.199

* median CER over substantive attempts only.

Bonus capability — grounding survives transcription fine-tuning: asked to locate a unique phrase on a page and answer with

markdown

{"bbox_2d": [x1, y1, x2, y2]}

(0–1000 normalized coordinates), v1.9a lands its box center inside the true phrase box on 58% of queries (chance ≈ 2–4%; median IoU 0.29) with a 100% JSON parse rate — despite zero grounding data in training.

Usage

python
import torch
from transformers import AutoImageProcessor, AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel

BASE = "Qwen/Qwen3-VL-8B-Instruct"
REV = "43e21bd7a6fedd323879fc5ea2c298df90a92784"  # step 1300

model = Qwen3VLForConditionalGeneration.from_pretrained(BASE, dtype=torch.bfloat16)
model = PeftModel.from_pretrained(
    model, "isaacmg/qwen3-vl-8b-hebrew-v19a-ckpt",
    subfolder="last-checkpoint", revision=REV)
processor = AutoProcessor.from_pretrained(BASE)
# REQUIRED: the training resolution policy
processor.image_processor = AutoImageProcessor.from_pretrained(
    BASE, min_pixels=6_500_000, max_pixels=7_000_000)

prompt = ("This is a manuscript from the Cairo Genizah. The text may be in "
          "Hebrew, Aramaic, or Judaeo-Arabic (Arabic written in Hebrew script).\n\n"
          "Transcribe the text in this image exactly as written. Do not "
          "normalize or correct the text.\nPreserve all vocalization marks "
          "(nikud) and line structure.\nMark damaged or unclear characters "
          "with [?].\n\nReturn ONLY the transcription with no commentary.")

For local inference the adapter can be merged into the official base (merge_and_unload()) and converted to MLX / GGUF; ship preprocessor_config.json with the pixel policy above alongside any export.

Limitations

Research model. Optimized for diplomatic transcription (copy what is visible); it is not a text normalizer, translator, or nikud restorer.
The hardest pages (heavy damage, dense two-column layouts) can still produce repetition loops or partial reads; per-page behavior can flip between checkpoints. Use the failure-mode-aware scoring above rather than bare CER when evaluating.
Trained at 6.5 MP: feeding low-resolution images, or overriding the processor policy, degrades it sharply.
Ground-truth transcriptions inherit scholarly conventions (damage markers [...], uncertainty [?]); the model reproduces those conventions.

Data sources & acknowledgements

Training transcriptions derive from scholarly editions and word-level manuscript annotations made available through the National Library of Israel ("KTIV" digitized manuscripts) and the Princeton Geniza Project, plus synthetic renders. This repository contains adapter weights only — no manuscript images and no transcription texts are redistributed here. Manuscript images remain under the terms of their holding institutions (Cambridge University Library, Bodleian Libraries, JTS, National Library of Israel, and others).

Base model: Qwen/Qwen3-VL-8B-Instruct (Apache-2.0).

The series at a glance

One line per generation — which checkpoint to use and which are historical:

Table with columns: version, repo, status, canonical revision, headline (benchmark)
version	repo	status	canonical revision	headline (benchmark)
v2.0a	v20a-ckpt	current best on both corpora; first with grounding	step 1800 `af9df6a0`	F1 0.849 (religious-140) / 0.875 (PGP-131); locate IoU 0.467, read_box CER 0.129
v1.9a	v19a-ckpt	FLAGSHIP — use this	step 1300

* CER over substantive attempts only, alignment-based scorer. Benchmarks differ across generations (Talmud print vs Genizah manuscripts) — compare within a row's named benchmark, not across rows.

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

isaacmg

Model Tree

Base

Qwen/Qwen3-VL-8B-Instruct

Adapter

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer