StyleInV: A Temporal Style Modulated Inversion Network for Unconditional
  Video Generation
Yuhan Wang Liming Jiang Chen Change Loy
Unconditional video generation is a challenging task that involves
synthesizing high-quality videos that are both coherent and of extended
duration. To address this challenge, researchers have used pretrained StyleGAN
image generators for high-quality frame synthesis and focused on motion
generator design. The motion generator is trained in an autoregressive manner
using heavy 3D convolutional discriminators to ensure motion coherence during
video generation. In this paper, we introduce a novel motion generator design
that uses a learning-based inversion network for GAN. The encoder in our method
captures rich and smooth priors from encoding images to latents, and given the
latent of an initially generated frame as guidance, our method can generate
smooth future latent by modulating the inversion encoder temporally. Our method
enjoys the advantage of sparse training and naturally constrains the generation
space of our motion generator with the inversion network guided by the initial
frame, eliminating the need for heavy discriminators. Moreover, our method
supports style transfer with simple fine-tuning when the encoder is paired with
a pretrained StyleGAN generator. Extensive experiments conducted on various
benchmarks demonstrate the superiority of our method in generating long and
high-resolution videos with decent single-frame quality and temporal
consistency.