TI-JEPA: An Innovative Energy-based Joint Embedding Strategy for Text-Image Multimodal Systems
It addresses the discrepancy problem in multimodal fusion for AI applications, potentially improving downstream tasks like visual question answering, but appears incremental as it builds on existing energy-based frameworks.
This paper tackled the semantic gap in text-image multimodal alignment by introducing TI-JEPA, an energy-based pre-training strategy, which achieved state-of-the-art performance on multimodal sentiment analysis benchmarks.
This paper focuses on multimodal alignment within the realm of Artificial Intelligence, particularly in text and image modalities. The semantic gap between the textual and visual modality poses a discrepancy problem towards the effectiveness of multi-modalities fusion. Therefore, we introduce Text-Image Joint Embedding Predictive Architecture (TI-JEPA), an innovative pre-training strategy that leverages energy-based model (EBM) framework to capture complex cross-modal relationships. TI-JEPA combines the flexibility of EBM in self-supervised learning to facilitate the compatibility between textual and visual elements. Through extensive experiments across multiple benchmarks, we demonstrate that TI-JEPA achieves state-of-the-art performance on multimodal sentiment analysis task (and potentially on a wide range of multimodal-based tasks, such as Visual Question Answering), outperforming existing pre-training methodologies. Our findings highlight the potential of using energy-based framework in advancing multimodal fusion and suggest significant improvements for downstream applications.