Edit As You Wish: Video Description Editing with Multi-grained Commands
Linli Yao Yuanmeng Zhang Ziheng Wang Xinglin Hou Tiezheng Ge Yuning Jiang Qin Jin
Automatically narrating a video with natural language can assist people in
grasping and managing massive videos on the Internet. From the perspective of
video uploaders, they may have varied preferences for writing the desired video
description to attract more potential followers, e.g. catching customers'
attention for product videos. The Controllable Video Captioning task is
therefore proposed to generate a description conditioned on the user demand and
video content. However, existing works suffer from two shortcomings: 1) the
control signal is fixed and can only express single-grained control; 2) the
video description can not be further edited to meet dynamic user demands. In
this paper, we propose a novel Video Description Editing (VDEdit) task to
automatically revise an existing video description guided by flexible user
requests. Inspired by human writing-revision habits, we design the user command
as a {operation, position, attribute} triplet to cover multi-grained use
requirements, which can express coarse-grained control (e.g. expand the
description) as well as fine-grained control (e.g. add specified details in
specified position) in a unified format. To facilitate the VDEdit task, we
first automatically construct a large-scale benchmark dataset namely VATEX-EDIT
in the open domain describing diverse human activities. Considering the
real-life application scenario, we further manually collect an e-commerce
benchmark dataset called EMMAD-EDIT. We propose a unified framework to convert
the {operation, position, attribute} triplet into a textual control sequence to
handle multi-grained editing commands. For VDEdit evaluation, we adopt
comprehensive metrics to measure three aspects of model performance, including
caption quality, caption-command consistency, and caption-video alignment.