Usage
The simplest way is the hawkeye package, which applies the exact protocol used in the paper:
pip install "hawkeye[infer] @ git+https://github.com/byron123t/hawkeye"
from hawkeye import Hawkeyeprint(Hawkeye("byron123t/Hawkeye-8B").read("label.jpg"))
With transformers directly (>= 4.57), reproduce the same protocol: an OCR system prompt, the
image upscaled ×3 with bicubic interpolation (the processor then caps it at 1,003,520 pixels),
the instruction before the image, greedy decoding.
import torchfrom PIL import Imagefrom transformers import AutoProcessor, Qwen3VLForConditionalGeneration SYSTEM_PROMPT = ("You are an OCR engine. Transcribe the text visible in the image exactly as it appears.\nOutput rules:\n- Output ONLY the transcribed text. No preamble, no explanation, no commentary, no closing remarks.\n- Do not describe the image. Do not say what the text is or where it is.\n- Preserve the original line breaks: one visual line of text per output line.\n- Do not wrap the output in quotes or code fences, and do not add Markdown.\n- Do not correct spelling or expand abbreviations; reproduce what is printed.\n- If no text is legible, output nothing.") model_id = "byron123t/Hawkeye-8B"model = Qwen3VLForConditionalGeneration.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")processor = AutoProcessor.from_pretrained(model_id) image = Image.open("label.jpg").convert("RGB")image = image.resize((image.width * 3, image.height * 3), Image.BICUBIC) # x3 bicubic messages = [ {"role": "system", "content": [{"type": "text", "text": SYSTEM_PROMPT}]}, {"role": "user", "content": [{"type": "text", "text": "Transcribe all the text in this image."}, {"type": "image", "image": image}]},]text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)out = model.generate(**inputs, max_new_tokens=4096, do_sample=False)print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
For exact reproduction of the reported numbers the evaluation also re-encodes each image as
JPEG (quality 92, longest side at most 2,048 px) before the ×3 upscale; hawkeye does this.
The shipped generation_config.json is greedy with up to 4,096 new tokens, as in the paper.
Results
Held-out BSTRB test split as in the paper (1,331 video frames from 224 videos, 241 photos, human-verified
labels; the public split omits one video for privacy, which changes these scores by at most 0.001). Differences are paired against the base model on the same items, 95% bootstrap CIs
clustered by video, Holm-corrected across all comparisons.
Table with columns: Hawkeye-8B vs. base Qwen3-VL-8B | Hawkeye-8B vs. base Qwen3-VL-8B |
|---|
| BSTRB video, BoW F1 | 0.675 → 0.744 (+6.9 pp, 95% CI [+5.8, +8.2]) |
| BSTRB photos, BoW F1 | +5.3 pp [+3.1, +7.6] |
| 10 general OCR benchmarks, BoW F1 | 2 of 10 better, 8 no significant change, 0 worse (HierText +6.4, MapText +4.4 pp) |
| Smart-glasses captures (37 objects, out of distribution) | word inclusion +13.0 pp, BoW F1 +11.5 pp, aligned F1 +12.3 pp (all significant; 26 of 37 objects improve, 5 regress) |
| Snellen charts, text ≤ 7 px (character accuracy) | +7.5 pp [+4.7, +10.2] |
| Repetition loops, video text ≤ 20 px | 8.5% → 5.2% |
Against the strongest API models (Qwen3-VL-235B, GPT-6 Astra, Kimi K2.6), Hawkeye 8B is on par over the full test split (BoW F1 0.730 against 0.731, 0.719 and 0.718; not significant) and significantly ahead on video frames with text at or below 20 px (+3.7 to +6.0 pp). On the pilot photos, which are excluded from training, the largest API models remain ahead (0.655 against 0.773 for GPT-6 Astra).
Training
- SFT on the BSTRB training video frames (6,734 frames from 1,133 videos; the public split omits 30 of them for privacy), vision encoder
unfrozen, learning rate 1e-6 (LLM and ViT) / 1e-5 (projector), inputs upscaled ×3.
- GRPO with LoRA (rank 64, alpha 128, on q/k/v/o; peak learning rate 2e-5), reward = bag-of-words F1 against the training
transcription, 300 steps; the LoRA is merged into the released weights.
Scripts and exact settings: github.com/byron123t/hawkeye.
Limitations
- Trained for English-dominant consumer packaging, signs, books and screens; non-Latin scripts
are rare in the training data.
- Training labels are GPT-5 transcriptions (human-verified labels exist only for val/test).
- The gain in precision mostly comes from fewer repetition loops; on outputs that do not loop,
the share of words not in the reference is similar to the base model's. Its most common
failure is stopping early on dense text.
- On very small or unreadable text it can produce plausible but wrong completions (e.g. nutrition
values); do not rely on it where errors matter (medication, legal text) without verification.
License
Apache-2.0, as is the base model. The BSTRB dataset used for training has its own license
(CC BY-NC 4.0, gated).
Citation
@article{tang2026hawkeye, title = {Hawkeye: Reading Illegible Text with Vision Language Models}, author = {Tang, Brian and Shin, Kang G.}, journal = {arXiv preprint arXiv:2610.41254}, year = {2026}}