Subject-Oriented Video Captioning
Yunchuan Ma Chang Teng Yuankai Qi Guorong Li Laiyu Qing Qi Wu Qingming Huang
Describing video content according to users' needs is a long-held goal.
Although existing video captioning methods have made significant progress, the
generated captions may not focus on the entity that users are particularly
interested in. To address this problem, we propose a new video captioning task,
subject-oriented video captioning, which allows users to specify the describing
target via a bounding box. To support this task, we construct two
subject-oriented video captioning datasets based on two widely used video
captioning datasets: MSVD and MSRVTT, by annotating subjects in each video for
each caption. These datasets pave the way for future technique development. As
the first attempt, we evaluate four state-of-the-art general video captioning
models, and have observed a large performance drop. We then explore several
strategies to enable them to describe the desired target. Experimental results
show obvious improvement, but there is still a large room for further
exploration in this field.