Edit3K: Universal Representation Learning for Video Editing Components
Xin Gu Libo Zhang Fan Chen Longyin Wen Yufei Wang Tiejian Luo Sijie Zhu
This paper focuses on understanding the predominant video creation pipeline,
i.e., compositional video editing with six main types of editing components,
including video effects, animation, transition, filter, sticker, and text. In
contrast to existing visual representation learning of visual materials (i.e.,
images/videos), we aim to learn visual representations of editing
actions/components that are generally applied on raw materials. We start by
proposing the first large-scale dataset for editing components of video
creation, which covers about $3,094$ editing components with $618,800$ videos.
Each video in our dataset is rendered by various image/video materials with a
single editing component, which supports atomic visual understanding of
different editing components. It can also benefit several downstream tasks,
e.g., editing component recommendation, editing component
recognition/retrieval, etc. Existing visual representation methods perform
poorly because it is difficult to disentangle the visual appearance of editing
components from raw materials. To that end, we benchmark popular alternative
solutions and propose a novel method that learns to attend to the appearance of
editing components regardless of raw materials. Our method achieves favorable
results on editing component retrieval/recognition compared to the alternative
solutions. A user study is also conducted to show that our representations
cluster visually similar editing components better than other alternatives.
Furthermore, our learned representations used to transition recommendation
tasks achieve state-of-the-art results on the AutoTransition dataset. The code
and dataset will be released for academic use.