Zoom and Shift are All You Need
This addresses multimodal learning challenges for researchers and practitioners, but appears incremental as it builds on existing feature alignment mechanisms.
The paper tackles the problem of feature alignment for multimodal data fusion by proposing an alternating shift-and-expand process to integrate information, achieving state-of-the-art results on tasks involving time series, images, and text.
Feature alignment serves as the primary mechanism for fusing multimodal data. We put forth a feature alignment approach that achieves full integration of multimodal information. This is accomplished via an alternating process of shifting and expanding feature representations across modalities to obtain a consistent unified representation in a joint feature space. The proposed technique can reliably capture high-level interplay between features originating from distinct modalities. Consequently, substantial gains in multimodal learning performance are attained. Additionally, we demonstrate the superiority of our approach over other prevalent multimodal fusion schemes on a range of tasks. Extensive experimental evaluation conducted on multimodal datasets comprising time series, image, and text demonstrates that our method achieves state-of-the-art results.