ART$\boldsymbol{\cdot}$V: Auto-Regressive Text-to-Video Generation with
  Diffusion Models
Wenming Weng Ruoyu Feng Yanhui Wang Qi Dai Chunyu Wang Dacheng Yin Zhiyuan Zhao Kai Qiu Jianmin Bao Yuhui Yuan Chong Luo Yueyi Zhang Zhiwei Xiong
We present ART$\boldsymbol{\cdot}$V, an efficient framework for
auto-regressive video generation with diffusion models. Unlike existing methods
that generate entire videos in one-shot, ART$\boldsymbol{\cdot}$V generates a
single frame at a time, conditioned on the previous ones. The framework offers
three distinct advantages. First, it only learns simple continual motions
between adjacent frames, therefore avoiding modeling complex long-range motions
that require huge training data. Second, it preserves the high-fidelity
generation ability of the pre-trained image diffusion models by making only
minimal network modifications. Third, it can generate arbitrarily long videos
conditioned on a variety of prompts such as text, image or their combinations,
making it highly versatile and flexible. To combat the common drifting issue in
AR models, we propose masked diffusion model which implicitly learns which
information can be drawn from reference images rather than network predictions,
in order to reduce the risk of generating inconsistent appearances that cause
drifting. Moreover, we further enhance generation coherence by conditioning it
on the initial frame, which typically contains minimal noise. This is
particularly useful for long video generation. When trained for only two weeks
on four GPUs, ART$\boldsymbol{\cdot}$V already can generate videos with natural
motions, rich details and a high level of aesthetic quality. Besides, it
enables various appealing applications, e.g., composing a long video from
multiple text prompts.