Cut-and-Paste: Subject-Driven Video Editing with Attention Control
Zhichao Zuo Zhao Zhang Yan Luo Yang Zhao Haijun Zhang Yi Yang Meng Wang
This paper presents a novel framework termed Cut-and-Paste for real-word
semantic video editing under the guidance of text prompt and additional
reference image. While the text-driven video editing has demonstrated
remarkable ability to generate highly diverse videos following given text
prompts, the fine-grained semantic edits are hard to control by plain textual
prompt only in terms of object details and edited region, and cumbersome long
text descriptions are usually needed for the task. We therefore investigate
subject-driven video editing for more precise control of both edited regions
and background preservation, and fine-grained semantic generation. We achieve
this goal by introducing an reference image as supplementary input to the
text-driven video editing, which avoids racking your brain to come up with a
cumbersome text prompt describing the detailed appearance of the object. To
limit the editing area, we refer to a method of cross attention control in
image editing and successfully extend it to video editing by fusing the
attention map of adjacent frames, which strikes a balance between maintaining
video background and spatio-temporal consistency. Compared with current
methods, the whole process of our method is like ``cut" the source object to be
edited and then ``paste" the target object provided by reference image. We
demonstrate that our method performs favorably over prior arts for video
editing under the guidance of text prompt and extra reference image, as
measured by both quantitative and subjective evaluations.