L-Verse: Bidirectional Generation Between Image and Text
This addresses the problem of cross-modal generation for vision-language tasks, offering a scalable and unified approach, though it appears incremental as it builds on existing transformer and VQ-VAE methods.
The paper tackles bidirectional generation between images and text by proposing L-Verse, a novel architecture combining AugVAE and BiART, which achieves state-of-the-art reconstruction on ImageNet1K and impressive results on MS-COCO Captions for both image-to-text and text-to-image generation without finetuning.
Far beyond learning long-range interactions of natural language, transformers are becoming the de-facto standard for many vision tasks with their power and scalability. Especially with cross-modal tasks between image and text, vector quantized variational autoencoders (VQ-VAEs) are widely used to make a raw RGB image into a sequence of feature vectors. To better leverage the correlation between image and text, we propose L-Verse, a novel architecture consisting of feature-augmented variational autoencoder (AugVAE) and bidirectional auto-regressive transformer (BiART) for image-to-text and text-to-image generation. Our AugVAE shows the state-of-the-art reconstruction performance on ImageNet1K validation set, along with the robustness to unseen images in the wild. Unlike other models, BiART can distinguish between image (or text) as a conditional reference and a generation target. L-Verse can be directly used for image-to-text or text-to-image generation without any finetuning or extra object detection framework. In quantitative and qualitative experiments, L-Verse shows impressive results against previous methods in both image-to-text and text-to-image generation on MS-COCO Captions. We furthermore assess the scalability of L-Verse architecture on Conceptual Captions and present the initial result of bidirectional vision-language representation learning on general domain.