MTVG : Multi-text Video Generation with Text-to-Video Models
Gyeongrok Oh Jaehwan Jeong Sieun Kim Wonmin Byeon Jinkyu Kim Sungwoong Kim Hyeokmin Kwon Sangpil Kim
Recently, video generation has attracted massive attention and yielded
noticeable outcomes. Concerning the characteristics of video, multi-text
conditioning incorporating sequential events is necessary for next-step video
generation. In this work, we propose a novel multi-text video generation~(MTVG)
by directly utilizing a pre-trained diffusion-based text-to-video~(T2V)
generation model without additional fine-tuning. To generate consecutive video
segments, visual consistency generated by distinct prompts is necessary with
diverse variations, such as motion and content-related transitions. Our
proposed MTVG includes Dynamic Noise and Last Frame Aware Inversion which
reinitialize the noise latent to preserve visual coherence between videos of
different prompts and prevent repetitive motion or contents. Furthermore, we
present Structure Guiding Sampling to maintain the global appearance across the
frames in a single video clip, where we leverage iterative latent updates
across the preceding frame. Additionally, our Prompt Generator allows for
arbitrary format of text conditions consisting of diverse events. As a result,
our extensive experiments, including diverse transitions of descriptions,
demonstrate that our proposed methods show superior generated outputs in terms
of semantically coherent and temporally seamless video.Video examples are
available in our project page: https://kuai-lab.github.io/mtvg-page.