Towards Modality Generalization: A Benchmark and Prospective Analysis
This work addresses the challenge of enabling multi-modal models to generalize to unseen modalities, which is crucial for robust AI applications, but it is incremental as it builds on existing generalization methods.
This paper tackles the problem of multi-modal models struggling with unseen modalities in real-world scenarios by introducing Modality Generalization (MG), defining weak and strong cases, and proposing a benchmark; experiments reveal limitations of existing methods and suggest future directions.
Multi-modal learning has achieved remarkable success by integrating information from various modalities, achieving superior performance in tasks like recognition and retrieval compared to uni-modal approaches. However, real-world scenarios often present novel modalities that are unseen during training due to resource and privacy constraints, a challenge current methods struggle to address. This paper introduces Modality Generalization (MG), which focuses on enabling models to generalize to unseen modalities. We define two cases: Weak MG, where both seen and unseen modalities can be mapped into a joint embedding space via existing perceptors, and Strong MG, where no such mappings exist. To facilitate progress, we propose a comprehensive benchmark featuring multi-modal algorithms and adapt existing methods that focus on generalization. Extensive experiments highlight the complexity of MG, exposing the limitations of existing methods and identifying key directions for future research. Our work provides a foundation for advancing robust and adaptable multi-modal models, enabling them to handle unseen modalities in realistic scenarios.