LAViTeR: Learning Aligned Visual and Textual Representations Assisted by Image and Caption Generation
This addresses the challenge of improving multimodal representation learning for downstream applications, though it appears incremental as it builds on existing transformer-based pre-training methods.
The paper tackles the problem of aligning visual and textual representations for vision-language tasks by proposing LAViTeR, which uses a Visual Textual Alignment module assisted by image synthesis and captioning tasks, resulting in superior alignment on CUB and MS-COCO datasets.
Pre-training visual and textual representations from large-scale image-text pairs is becoming a standard approach for many downstream vision-language tasks. The transformer-based models learn inter and intra-modal attention through a list of self-supervised learning tasks. This paper proposes LAViTeR, a novel architecture for visual and textual representation learning. The main module, Visual Textual Alignment (VTA) will be assisted by two auxiliary tasks, GAN-based image synthesis and Image Captioning. We also propose a new evaluation metric measuring the similarity between the learnt visual and textual embedding. The experimental results on two public datasets, CUB and MS-COCO, demonstrate superior visual and textual representation alignment in the joint feature embedding space