TengfeiLiuCoder

RefCaptioner

README

License: apache-2.0

Task overview

Multi-reference image-grounded video captioning

RefCaptioner grounds relevant reference images to local phrases in a video caption while rejecting distractor references.

Model details

Table with columns: Item, Value
Item	Value
Base model	`Qwen/Qwen3-VL-8B-Instruct`
Release checkpoint	`RefCaptioner1.0`
Input	Ordered reference images plus one video
Output	English caption with local `<Image_n>` bindings
Thinking mode	Disabled
Weight format	Merged `safetensors`

Method overview

RefCaptioner training framework Training Framework

Installation

The tested environment for loading RefCaptioner and running inference is provided in requirement.txt. Use Python 3.10 and install it with:

bash
pip install -r requirement.txt

Quick start

bash
python inference.py \
  --model TengfeiLiuCoder/RefCaptioner \
  --video /path/to/video.mp4 \
  --images /path/to/ref_1.jpg /path/to/ref_2.jpg /path/to/ref_3.png

The image order defines the tag mapping: the first image is <Image_1>, the second is <Image_2>, and so on. The script uses the exact prompt structure and deterministic decoding configuration used for the reported test-set inference.

Reproduction configuration

The released benchmark captions were generated with the following settings:

Table with columns: Parameter, Value
Parameter	Value
Prompt protocol	`Prompt_1.0`
Maximum input length	18,000 tokens
Maximum new tokens	512
Video sampling rate	2 FPS
Minimum video frames	4
Reference-image maximum pixels	602,112
Video-frame maximum pixels	602,112
Decoding	Greedy (`do_sample=False`)

The same values are machine-readable in inference_config.json. The eight GPUs used for the full benchmark only provided sample-level parallelism; eight GPUs are not a model requirement. A single device can run the model when it has enough memory, while device_map="auto" can distribute the model across available devices.

Input and output contract

The prompt labels each ordered reference before presenting the associated image, then presents the video. The model must:

describe visible video content in one English paragraph;
place every used <Image_n> immediately after a concrete grounded phrase;
group tags only when they ground the same visual unit; and
omit distractor or uncertain references rather than forcing all tags into the caption.

Example output:

RefCaptioner example output

MRVBench Results

The following scores report RefCaptioner on MRVBench. Eff-Bind = Ref-Tag-R × Ref-Bind.

Table with columns: Model, KP-Cov ↑, VQA ↑, VQA-Cov ↑, Ref-Tag-P ↑, Ref-Tag-R ↑, Ref-Bind ↑, Eff-Bind ↑, Dist-Rej ↑, FalseRef-Any ↓, Subj-R ↑, Subj-F1 ↑, MRVScore ↑
Model	KP-Cov ↑	VQA ↑	VQA-Cov ↑	Ref-Tag-P ↑	Ref-Tag-R ↑	Ref-Bind ↑	Eff-Bind ↑	Dist-Rej ↑	FalseRef-Any ↓	Subj-R ↑	Subj-F1 ↑	MRVScore ↑
RefCaptioner	0.882	0.686	0.837	0.994	0.943

These values are tied to the prompt and preprocessing configuration above; changing frame sampling, pixel limits, prompt wording, or decoding can change the results.

Intended use

The model is intended for research on multi-reference video understanding, grounded caption generation, reference selection, and local phrase-to-image binding. It is not designed to identify people, infer sensitive attributes, or establish the provenance or ownership of visual content.

Limitations

Results depend on frame sampling and may miss short events between sampled frames.
The output is English and follows a task-specific caption style.
Reference tags express visual grounding, not identity verification.

Users should verify generated descriptions before applying them in consequential settings.

License

The model weights are released under the Apache 2.0 license, following the base model. The companion benchmark contains separate media and annotation considerations; consult its dataset card before use or redistribution.

Citation

bibtex
@misc{liu2026refcaptionermultireferenceimagegroundedvideo,
      title={RefCaptioner: Multi-Reference Image-Grounded Video Captioning},
      author={Tengfei Liu and Yang Shi and Yuran Wang and Xiaohan Zhang and Yuqing Wen and Yuqi Tang and Qixun Wang and Zhuoran Zhang and Xuanyu Zhu and Weihong Lin and Xinlei Yu and Yujie Wei and Xinwei Long and Fengxiang Wang and Xinlong Chen and Yue Ding and Jialu Chen and Haotian Wang and Yuanxing Zhang},
      year={2026},
      eprint={2607.28509},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.28509}
}

Available on FriendliAI

Dedicated Endpoints

Run this model inference on single tenant GPU with unmatched speed and reliability at scale.

Learn more

Container

Run this model inference with full control and performance in your environment.

Learn more

Model Details

Model Provider

TengfeiLiuCoder

Model Tree

Base

Qwen/Qwen3-VL-8B-Instruct

Fine-tuned

this model

Input Modalities

TextImage

Output Modalities

Text

Supported Functionality

Dedicated EndpointsContainer

Explore FriendliAI today

Get started Talk to an engineer