Grid Diffusion Models for Text-to-Video Generation
Taegyeong Lee Soyeong Kwon Taehwan Kim
Recent advances in the diffusion models have significantly improved
text-to-image generation. However, generating videos from text is a more
challenging task than generating images from text, due to the much larger
dataset and higher computational cost required. Most existing video generation
methods use either a 3D U-Net architecture that considers the temporal
dimension or autoregressive generation. These methods require large datasets
and are limited in terms of computational costs compared to text-to-image
generation. To tackle these challenges, we propose a simple but effective novel
grid diffusion for text-to-video generation without temporal dimension in
architecture and a large text-video paired dataset. We can generate a
high-quality video using a fixed amount of GPU memory regardless of the number
of frames by representing the video as a grid image. Additionally, since our
method reduces the dimensions of the video to the dimensions of the image,
various image-based methods can be applied to videos, such as text-guided video
manipulation from image manipulation. Our proposed method outperforms the
existing methods in both quantitative and qualitative evaluations,
demonstrating the suitability of our model for real-world video generation.