InstructVid2Vid: Controllable Video Editing with Natural Language
  Instructions
Bosheng Qin Juncheng Li Siliang Tang Tat-Seng Chua Yueting Zhuang
We present an end-to-end diffusion-based method for editing videos with human
language instructions, namely $\textbf{InstructVid2Vid}$. Our approach enables
the editing of input videos based on natural language instructions without any
per-example fine-tuning or inversion. The proposed InstructVid2Vid model
combines a pretrained image generation model, Stable Diffusion, with a
conditional 3D U-Net architecture to generate time-dependent sequence of video
frames. To obtain the training data, we incorporate the knowledge and expertise
of different models, including ChatGPT, BLIP, and Tune-a-Video, to synthesize
video-instruction triplets, which is a more cost-efficient alternative to
collecting data in real-world scenarios. To improve the consistency between
adjacent frames of generated videos, we propose the Frame Difference Loss,
which is incorporated during the training process. During inference, we extend
the classifier-free guidance to text-video input to guide the generated
results, making them more related to both the input video and instruction.
Experiments demonstrate that InstructVid2Vid is able to generate high-quality,
temporally coherent videos and perform diverse edits, including attribute
editing, change of background, and style transfer. These results highlight the
versatility and effectiveness of our proposed method. Code is released in
$\href{https://github.com/BrightQin/InstructVid2Vid}{InstructVid2Vid}$.