A Simple Text to Video Model via Transformer
This addresses video generation from text, an incremental improvement in multimodal AI for content creation applications.
The authors tackled text-to-video generation by proposing a Transformer-based model that encodes text and images into a shared hidden space, using U-Net to handle image signal degradation in long sequences and adding motion constraints. They tested on UCF101 with GPT2, reporting promising results but without specific quantitative metrics.
We present a general and simple text to video model based on Transformer. Since both text and video are sequential data, we encode both texts and images into the same hidden space, which are further fed into Transformer to capture the temporal consistency and then decoder to generate either text or images. Considering the image signal may become weak in the long sequence, we introduce the U-Net to reconstruct image from its noised version. Specifically, we increase the noise level to the original image in the long sequence, then use the $down$ module from U-Net to encode noised images, which are further input to transformer to predict next clear images. We also add a constraint to promote motion between any generated image pair in the video. We use GPT2 and test our approach on UCF101 dataset and show it can generate promising videos.