Results
Table with columns: Model, Families, Questions, Macro accuracy %, Micro accuracy %, Mean family ECE % ↓| Model | Families | Questions | Macro accuracy % | Micro accuracy % | Mean family ECE % ↓ |
|---|
| Base 0.8B | 30 | 21,456 | 40.07 | 40.04 | 20.83 |
| SFT 0.8B | 30 | 41,951 | 47.86 | 48.09 | — |
| v1.1 0.8B | 30 | 41,975 | 64.52 | 65.40 | 4.30 |
| Base 2B | 30 | 21,456 | 36.30 | 35.94 | 34.21 |
| v1.1 2B | 30 | 41,975 | 64.46 | 65.42 | 6.27 |
| Base 4B | 30 | 21,456 | 40.12 | 39.79 | 30.04 |
| v1.1 4B | 30 | 41,975 | 70.00 | 70.82 | 5.89 |
Table with columns: Family, Base 0.8B, SFT 0.8B, v1.1 0.8B, Base 2B, v1.1 2B, Base 4B, v1.1 4B| Family | Base 0.8B | SFT 0.8B | v1.1 0.8B | Base 2B | v1.1 2B | Base 4B | v1.1 4B |
|---|
| androidcontrol | 29.49 | 40.73 | 71.70 | 21.54 | 71.84 | 28.81 | 77.30 |
| atari | 45.68 | 45.07 | 71.67 |
Protocol and limitations
The table reports existing evaluations, not a matched-sample ablation. Base models use candidate-answer probabilities (A1_raw), with at most 1,000 questions per family before dev/test splitting. OmniJev and SFT typically use about 1,500 questions per family. Sampling, option truncation and image inputs differ. The 30-family summary excludes music, for which only base results exist.
The released SFT baseline is 0.8B only. Its 41,951 generations were judged and audited: 97 incorrect explicit answer labels were accepted by the raw LLM judge. Audited accuracy is 48.0894% (equal to the parser), rather than the uncorrected judge's 48.3207%.
v1.1 OmniJev uses numbered multi-image panels. The historical SFT and base evaluations used the first still image only. Multi-image SFT retraining and inference over the newly frozen 272,561-question set are not completed. The 20-step distributed smoke checkpoint is not part of this release.
These are project evaluation families, not uniform official benchmark scores. POPE and LongVideoBench were excluded from training; Mind2Web uses official splits. Other families include custom row-level holdouts; the event-timing family uses videos from the Charades-STA training set. Row-level exclusion does not establish episode-level separation.
The main table contains raw evaluation reports, not a new evaluation of the calibrated serving outputs. Released temperature metadata comes from a separate calibration split. The 4B checkpoint also contains biases.noul=0.0531085661, which moves the yes/no boundary; therefore its raw POPE result must not be restated as a serving-path result.
4B has the highest 30-family macro accuracy, but POPE (84.73% vs 86.42% base) and LVB (56.80% vs 59.02% base) remain lower. The 0.8B and 2B models also have substantial POPE regressions. All rows are retained.
License and attribution
Apache-2.0 for the adapter and project code; the Qwen backbone retains its own license. Developed by Beijing Zhongguancun Academy, the Institute of Automation, Chinese Academy of Sciences, and Zevo.