What it does
Given one video frame, the model lists the objects the person in the frame is related to and describes each relation using the Action Genome label set. It answers with JSON only:
[{"object": "box", "attention": ["looking_at"], "spatial": ["in_front_of"], "contact": ["holding"]}]
attention: whether the person is looking at the object (looking_at, not_looking_at, unsure)
spatial: position relative to the person (e.g. in_front_of, beneath, behind, on_the_side_of)
contact: physical interaction (e.g. holding, sitting_on, carrying, touching, not_contacting)
The prompt used in training and evaluation (use it verbatim for best results):
List every visible object that the person is related to in this frame. Output JSON only: [{"object":<name>,"attention":[..],"spatial":[..],"contact":[..]}]
Training
Table | |
|---|
| Base model | LLM_MARKDOWN_PROTECTED_17 |
| Method | LoRA, bf16 (no quantisation), vision tower frozen |
| LoRA | r=16, alpha=32, dropout=0.05, on the language model's q/k/v/o/gate/up/down projections |
| Data | Action Genome train split: 3,922 frames from 500 Charades videos (up to 8 random annotated frames per video, resized to a short side of 448 px) |
| Objective | Next-token loss on the JSON answer only (the prompt is masked) |
| Schedule | 2 epochs (492 steps), lr 1e-4, cosine with 3% warmup, batch 2 x 8 accumulation steps = 16 |
| Hardware / time | One 16 GB laptop GPU (RTX A5000), about 32 minutes |
| Final average train loss | 0.183 (1.30 at step 10, about 0.14 by the end of epoch 1) |
Evaluation
Held-out Action Genome test-split frames: 199 frames from 25 videos. A prediction counts as correct only if the (object, relation) pair matches the ground truth exactly, for the attention and contact groups. The spatial group is not scored. Greedy decoding.
Table with columns: Model, JSON valid, Recall, Precision| Model | JSON valid | Recall | Precision |
|---|
| Qwen3-VL-2B-Instruct (no fine-tuning, same prompt) | 195 / 199 | 0.00 | 0.00 |
| This model | 199 / 199 | 0.31 | 0.45 |
The untuned model scores 0.00 under this exact-match scoring most likely because it does not use Action Genome's label words. I did not inspect its raw outputs to confirm that, so read the baseline as "does not follow the label scheme", not as "cannot see the scene".
Per-relation results for this model (n = number of ground-truth instances):
Table with columns: Relation, Recall, Precision, n| Relation | Recall | Precision | n |
|---|
| attention: not_looking_at | 0.36 | 0.50 | 245 |
| contact: holding | 0.39 | 0.55 | 200 |
| attention: looking_at | 0.29 | 0.34 | 150 |
| contact: not_contacting | 0.16 | 0.27 | 140 |
| attention: unsure | 0.11 | 0.35 | 64 |
| contact: touching | 0.29 | 0.42 | 48 |
| contact: sitting_on | 0.61 | 0.62 | 38 |
| contact: carrying | 0.67 | 0.86 | 9 |
| contact: standing_on | 0.00 | 0.00 | 5 |
Rows with small n (below about 15) are noisy.
Limitations
- Small, narrow training set. Trained on about 3.9k frames from indoor Charades videos. Expect weaker results on other settings (outdoors, sports, crowds). This was not tested.
- Modest accuracy. Recall 0.31 means most true relations are still missed.
not_contacting, unsure and rare contact classes are weak.
- Small evaluation. 199 frames from 25 videos, one run, no confidence intervals. The spatial group was not scored.
- Single frame. No temporal reasoning across frames.
- Closed vocabulary. It tends to answer with Action Genome's labels and objects only.
- Not a general scene-graph model. It only describes the person's relations, not relations between other objects.
License and intended use
The base model is Apache-2.0. The training data is Action Genome, built on the Charades videos, which are released for non-commercial research use. Treat this model as intended for research and education; check the Action Genome and Charades terms before any commercial use. This is a student/learning project, not a validated production system.
Acknowledgements
- Base model: Qwen3-VL-2B-Instruct (Alibaba Qwen team)
- Data: Action Genome (Ji et al., CVPR 2020), built on Charades (Sigurdsson et al., ECCV 2016)
How to use
import json, re, torchfrom PIL import Imagefrom transformers import AutoModelForImageTextToText, AutoProcessor repo = "Ram20307/qwen3vl-2b-ag-merged"proc = AutoProcessor.from_pretrained(repo)model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16, device_map="auto").eval() PROMPT = ('List every visible object that the person is related to in this frame. Output JSON only: ' '[{"object":<name>,"attention":[..],"spatial":[..],"contact":[..]}]')msgs = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": PROMPT}]}]text = proc.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True)x = proc(text=[text], images=[Image.open("frame.jpg").convert("RGB")], return_tensors="pt").to(model.device)with torch.no_grad(): y = model.generate(**x, max_new_tokens=400, do_sample=False)out = proc.batch_decode(y[:, x["input_ids"].shape[1]:], skip_special_tokens=True)[0]print(json.loads(re.search(r"\[.*\]", out, re.S).group(0)))
The merged weights were produced with PeftModel.merge_and_unload(). They have not yet been re-evaluated on their own; the 4 GB repo is meant for convenience and as a starting point for GGUF/quantised conversion.