MovieFactory: Automatic Movie Creation from Text using Large Generative
  Models for Language and Images
Junchen Zhu Huan Yang Huiguo He Wenjing Wang Zixi Tuo Wen-Huang Cheng Lianli Gao Jingkuan Song Jianlong Fu
In this paper, we present MovieFactory, a powerful framework to generate
cinematic-picture (3072$\times$1280), film-style (multi-scene), and
multi-modality (sounding) movies on the demand of natural languages. As the
first fully automated movie generation model to the best of our knowledge, our
approach empowers users to create captivating movies with smooth transitions
using simple text inputs, surpassing existing methods that produce soundless
videos limited to a single scene of modest quality. To facilitate this
distinctive functionality, we leverage ChatGPT to expand user-provided text
into detailed sequential scripts for movie generation. Then we bring scripts to
life visually and acoustically through vision generation and audio retrieval.
To generate videos, we extend the capabilities of a pretrained text-to-image
diffusion model through a two-stage process. Firstly, we employ spatial
finetuning to bridge the gap between the pretrained image model and the new
video dataset. Subsequently, we introduce temporal learning to capture object
motion. In terms of audio, we leverage sophisticated retrieval models to select
and align audio elements that correspond to the plot and visual content of the
movie. Extensive experiments demonstrate that our MovieFactory produces movies
with realistic visuals, diverse scenes, and seamlessly fitting audio, offering
users a novel and immersive experience. Generated samples can be found in
YouTube or Bilibili (1080P).