Supervised Multimodal Bitransformers for Classifying Images and Text
This work addresses the need for improved multimodal classification in AI applications, representing an incremental advancement by adapting transformer models to handle both text and images.
The authors tackled the problem of multimodal classification by introducing a supervised multimodal bitransformer that fuses text and image encoders, achieving state-of-the-art performance on various benchmark tasks, including hard test sets designed for multimodal evaluation.
Self-supervised bidirectional transformer models such as BERT have led to dramatic improvements in a wide variety of textual classification tasks. The modern digital world is increasingly multimodal, however, and textual information is often accompanied by other modalities such as images. We introduce a supervised multimodal bitransformer model that fuses information from text and image encoders, and obtain state-of-the-art performance on various multimodal classification benchmark tasks, outperforming strong baselines, including on hard test sets specifically designed to measure multimodal performance.