Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
This work addresses the limitation of task-specific visual representations in vision-language tasks, potentially benefiting researchers and practitioners in multimodal AI by reducing annotation costs and improving accuracy.
The authors tackled the problem of aligning image pixels with text semantics directly, without relying on region-based features, by proposing Pixel-BERT, a deep multi-modal transformer model. Their approach achieved state-of-the-art results in downstream tasks, such as boosting VQA performance by 2.17 points compared to previous SOTA.
We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of bounding box annotations and overcomes the unbalance between semantic labels in visual task and language semantic. To provide a better representation for down-stream tasks, we pre-train a universal end-to-end model with image and sentence pairs from Visual Genome dataset and MS-COCO dataset. We propose to use a random pixel sampling mechanism to enhance the robustness of visual representation and to apply the Masked Language Model and Image-Text Matching as pre-training tasks. Extensive experiments on downstream tasks with our pre-trained model show that our approach makes the most state-of-the-arts in downstream tasks, including Visual Question Answering (VQA), image-text retrieval, Natural Language for Visual Reasoning for Real (NLVR). Particularly, we boost the performance of a single model in VQA task by 2.17 points compared with SOTA under fair comparison.