Generative Modeling of Class Probability for Multi-Modal Representation Learning
This addresses modality misalignment issues in multi-modal AI, offering improved generalization, but it is incremental as it builds on existing generative and alignment techniques.
The paper tackles the problem of modality discrepancies in multi-modal representation learning by proposing a class anchor alignment approach that uses class probability distributions, resulting in significant outperformance of state-of-the-art methods on four benchmark datasets, particularly in out-of-domain evaluations.
Multi-modal understanding plays a crucial role in artificial intelligence by enabling models to jointly interpret inputs from different modalities. However, conventional approaches such as contrastive learning often struggle with modality discrepancies, leading to potential misalignments. In this paper, we propose a novel class anchor alignment approach that leverages class probability distributions for multi-modal representation learning. Our method, Class-anchor-ALigned generative Modeling (CALM), encodes class anchors as prompts to generate and align class probability distributions for each modality, enabling more effective alignment. Furthermore, we introduce a cross-modal probabilistic variational autoencoder to model uncertainty in the alignment, enhancing the ability to capture deeper relationships between modalities and data variations. Extensive experiments on four benchmark datasets demonstrate that our approach significantly outperforms state-of-the-art methods, especially in out-of-domain evaluations. This highlights its superior generalization capabilities in multi-modal representation learning.