A Simple Text to Video Model via Transformer
Gang Chen
We present a general and simple text to video model based on Transformer.
Since both text and video are sequential data, we encode both texts and images
into the same hidden space, which are further fed into Transformer to capture
the temporal consistency and then decoder to generate either text or images.
Considering the image signal may become weak in the long sequence, we introduce
the U-Net to reconstruct image from its noised version. Specifically, we
increase the noise level to the original image in the long sequence, then use
the $down$ module from U-Net to encode noised images, which are further input
to transformer to predict next clear images. We also add a constraint to
promote motion between any generated image pair in the video. We use GPT2 and
test our approach on UCF101 dataset and show it can generate promising videos.