ViP-Mixer: A Convolutional Mixer for Video Prediction
Xin Zheng Ziang Peng Yuan Cao Hongming Shan Junping Zhang
Video prediction aims to predict future frames from a video's previous
content. Existing methods mainly process video data where the time dimension
mingles with the space and channel dimensions from three distinct angles: as a
sequence of individual frames, as a 3D volume in spatiotemporal coordinates, or
as a stacked image where frames are treated as separate channels. Most of them
generally focus on one of these perspectives and may fail to fully exploit the
relationships across different dimensions. To address this issue, this paper
introduces a convolutional mixer for video prediction, termed ViP-Mixer, to
model the spatiotemporal evolution in the latent space of an autoencoder. The
ViP-Mixers are stacked sequentially and interleave feature mixing at three
levels: frames, channels, and locations. Extensive experiments demonstrate that
our proposed method achieves new state-of-the-art prediction performance on
three benchmark video datasets covering both synthetic and real-world
scenarios.