BrokenBind: Universal Modality Exploration beyond Dataset Boundaries
This addresses a key bottleneck in multi-modal learning for AI researchers, allowing more flexible and cost-effective model training across diverse datasets, though it appears incremental as it builds on existing binding strategies.
The paper tackles the problem of multi-modal learning being limited to modalities within a single dataset, which hinders generalization and increases data acquisition costs. It introduces BrokenBind, a method that binds modalities from different datasets by generating pseudo embeddings, enabling universal modality exploration and showing effectiveness in low-data regimes.
Multi-modal learning combines various modalities to provide a comprehensive understanding of real-world problems. A common strategy is to directly bind different modalities together in a specific joint embedding space. However, the capability of existing methods is restricted within the modalities presented in the given dataset, thus they are biased when generalizing to unpresented modalities in downstream tasks. As a result, due to such inflexibility, the viability of previous methods is seriously hindered by the cost of acquiring multi-modal datasets. In this paper, we introduce BrokenBind, which focuses on binding modalities that are presented from different datasets. To achieve this, BrokenBind simultaneously leverages multiple datasets containing the modalities of interest and one shared modality. Though the two datasets do not correspond to each other due to distribution mismatch, we can capture their relationship to generate pseudo embeddings to fill in the missing modalities of interest, enabling flexible and generalized multi-modal learning. Under our framework, any two modalities can be bound together, free from the dataset limitation, to achieve universal modality exploration. Further, to reveal the capability of our method, we study intensified scenarios where more than two datasets are needed for modality binding and show the effectiveness of BrokenBind in low-data regimes. Through extensive evaluation, we carefully justify the superiority of BrokenBind compared to well-known multi-modal baseline methods.