NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion
This addresses the challenge of creating and editing visual content for applications in AI and media, representing a novel method rather than an incremental improvement.
The paper tackles the problem of generating and manipulating visual data across multiple tasks by introducing NÜWA, a unified multimodal pre-trained model, which achieves state-of-the-art results on tasks like text-to-image and text-to-video generation and demonstrates strong zero-shot capabilities.
This paper presents a unified multimodal pre-trained model called NÜWA that can generate new or manipulate existing visual data (i.e., images and videos) for various visual synthesis tasks. To cover language, image, and video at the same time for different scenarios, a 3D transformer encoder-decoder framework is designed, which can not only deal with videos as 3D data but also adapt to texts and images as 1D and 2D data, respectively. A 3D Nearby Attention (3DNA) mechanism is also proposed to consider the nature of the visual data and reduce the computational complexity. We evaluate NÜWA on 8 downstream tasks. Compared to several strong baselines, NÜWA achieves state-of-the-art results on text-to-image generation, text-to-video generation, video prediction, etc. Furthermore, it also shows surprisingly good zero-shot capabilities on text-guided image and video manipulation tasks. Project repo is https://github.com/microsoft/NUWA.