Results
Decision Index 0.3, public vision suite (12,210 rows, all answered), official kit:
Table with columns: Benchmark, jev-vision-27b (base), Herbert v0.5| Benchmark | jev-vision-27b (base) | Herbert v0.5 |
|---|
| CV-Bench | 76.8 | 79.2 |
| BLINK | 54.6 | 72.8 |
| RealWorldQA | 72.8 | 73.4 |
| CharXiv | 78.9 | 77.4 |
| InfographicVQA | 96.2 | 96.7 |
| Mind2Web | 84.4 | 83.1 |
| Winoground | 83.5 | 84.0 |
| KIE (CORD + FUNSD) | 91.3 | 92.5 |
| Hateful Memes | 52.0 | 52.7 |
| R-Bench-M | 24.7 | 31.2 |
| MMMU-Pro vision | 38.0 | 42.3 |
| Public vision index | 73.06 | 75.69 |
Run files: leinenterprises/herbert-results.
How it answers
Options are labelled A, B, C, ... in the prompt; the answer is read from the label logits at the answer position
and softmaxed over the listed options only. Up to 624 options per question (A-Z, a-z, 0-9, then single-token
two-letter labels); larger questions are refused as unsupported, never truncated. Images up to 8 per request.
Run it
pip install vllm fastapi uvicorn peft transformers "decision-index[vision] @ git+https://github.com/apolinario/decision-index"cd codepython -m train.merge --ckpt .. --out ../merged # adapter -> plain bf16 model (one-time)python -m serve.server --model ../merged --perms 1 --port 8000 --max-model-len 65536 --max-pixels 1638400
POST /v1/systemone (Decision Index wire format) or /v1/decisions (the kit's vision decide engine, images as a
separate list).
Training
- Starts from the jev-vision-27b adapter (LoRA r=32, alpha=64 on the language model; vision tower frozen).
- Added data: synthetic decision questions generated by our own renderer (3D scenes with depth, point
correspondence across views, camera motion, spatial relations and counts; charts drawn from known data; exam-style
pages with computed answers), labels exact by construction. No benchmark items and no evaluation splits were used.
- The previous mix (text decision corpus plus train-split vision data, see the base model's card) is replayed
alongside it.
Limits
Public-suite numbers are self-run with the official kit; the board's Full score also includes private tests run by the
maintainers.