LOVECon: Text-driven Training-Free Long Video Editing with ControlNet
Zhenyi Liao Zhijie Deng
Leveraging pre-trained conditional diffusion models for video editing without
further tuning has gained increasing attention due to its promise in film
production, advertising, etc. Yet, seminal works in this line fall short in
generation length, temporal coherence, or fidelity to the source video. This
paper aims to bridge the gap, establishing a simple and effective baseline for
training-free diffusion model-based long video editing. As suggested by prior
arts, we build the pipeline upon ControlNet, which excels at various image
editing tasks based on text prompts. To break down the length constraints
caused by limited computational memory, we split the long video into
consecutive windows and develop a novel cross-window attention mechanism to
ensure the consistency of global style and maximize the smoothness among
windows. To achieve more accurate control, we extract the information from the
source video via DDIM inversion and integrate the outcomes into the latent
states of the generations. We also incorporate a video frame interpolation
model to mitigate the frame-level flickering issue. Extensive empirical studies
verify the superior efficacy of our method over competing baselines across
scenarios, including the replacement of the attributes of foreground objects,
style transfer, and background replacement. In particular, our method manages
to edit videos with up to 128 frames according to user requirements. Code is
available at https://github.com/zhijie-group/LOVECon.