SAVE: Protagonist Diversification with Structure Agnostic Video Editing
Yeji Song Wonsik Shin Junsoo Lee Jeesoo Kim Nojun Kwak
Driven by the upsurge progress in text-to-image (T2I) generation models,
text-to-video (T2V) generation has experienced a significant advance as well.
Accordingly, tasks such as modifying the object or changing the style in a
video have been possible. However, previous works usually work well on trivial
and consistent shapes, and easily collapse on a difficult target that has a
largely different body shape from the original one. In this paper, we spot the
bias problem in the existing video editing method that restricts the range of
choices for the new protagonist and attempt to address this issue using the
conventional image-level personalization method. We adopt motion
personalization that isolates the motion from a single source video and then
modifies the protagonist accordingly. To deal with the natural discrepancy
between image and video, we propose a motion word with an inflated textual
embedding to properly represent the motion in a source video. We also regulate
the motion word to attend to proper motion-related areas by introducing a novel
pseudo optical flow, efficiently computed from the pre-calculated attention
maps. Finally, we decouple the motion from the appearance of the source video
with an additional pseudo word. Extensive experiments demonstrate the editing
capability of our method, taking a step toward more diverse and extensive video
editing.