Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-05
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 16
- gradient_accumulation_steps: 10
- total_train_batch_size: 320
- total_eval_batch_size: 128
- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.05
- num_epochs: 5.0
Training results
Framework versions
- Transformers 4.57.1
- Pytorch 2.8.0+cu128
- Datasets 4.0.0
- Tokenizers 0.22.2
October 2026 held-out evaluation
The final five-epoch model was evaluated on the preserved October test split: 9,522 examples (4,761 Yes and 4,761 No), 1,069 videos, and 244 tasks, with no October train/test video overlap. The data producer caps held-out examples per task; this is not a random 20% of the full expanded dataset.
- Answer accuracy: 80.81%
- Yes recall: 68.03%
- No recall: 93.59%
- Task-macro average precision: 91.64%
- Task-macro pairwise ranking accuracy: 91.43%
- Invalid answers: 0
Answer accuracy uses the first freely predicted token. Ranking scores use full-vocabulary Yes probability; pairs follow the original exported pairing and ties are errors. Sampling matches training: 8 fps, at most 128 frames, 65,536 pixels per frame, qwen3_vl_nothink, BF16, and SDPA. All examples retained their original order and fit below the 8,192-token cutoff. Distributed sampler padding was verified and excluded.
Historical comparison uses 7,020 current-label examples across 797 videos present in the retained February test file and absent from its rare-label training file (2,593 complete pairs). The full October test includes 255 videos used by the historical rare-label stage; use the shared subset for that comparison. The original complete February training JSON is unavailable, so historical held-out membership relies on the retained test file and full-test historical comparisons may include training exposure.
Table with columns: Model, Shared answer accuracy, Shared mAP, Shared pairwise accuracy| Model | Shared answer accuracy | Shared mAP | Shared pairwise accuracy |
|---|
| october | 81.18% | 92.44% | 92.23% |
| february | 80.54% | 91.42% | 90.84% |
| february_rare | 80.21% | 90.67% | 88.81% |
The October-minus-February shared-test accuracy difference has a 95% paired video-bootstrap interval of -0.40 to 1.69 percentage points, so its small accuracy advantage is not conclusive.
Evaluated model revision: fc27ebd93dd703070e734a4f36520a0d916cc9cd. See evaluation_comparison.json for exact model revisions, metrics, and paired video-bootstrap confidence intervals. Only aggregate evaluation files and this model card were updated; weights and training assets are unchanged.