Towards Achieving Perfect Multimodal Alignment
This work addresses the challenge of aligning different data modalities for representation learning, which is incremental as it builds on existing alignment methods with a new theoretical formulation.
The paper tackles the problem of multimodal alignment by formulating it as an inverse problem to achieve perfect alignment where paired data map to equivalent latent vectors, and shows that this method significantly enhances accuracy in cross-modal transfer for human action recognition compared to learned contrastive alignment.
Multimodal alignment constructs a joint latent vector space where modalities representing the same concept map to neighboring latent vectors. We formulate this as an inverse problem and show that, under certain conditions, paired data from each modality can map to equivalent latent vectors, which we refer to as perfect alignment. When perfect alignment cannot be achieved, it can be approximated using the Singular Value Decomposition (SVD) of a multimodal data matrix. Experiments on synthetic multimodal Gaussian data verify the effectiveness of our perfect alignment method compared to a learned contrastive alignment method. We further demonstrate the practical application of cross-modal transfer for human action recognition, showing that perfect alignment significantly enhances the model's accuracy. We conclude by discussing how these findings can be applied to various modalities and tasks and the limitations of our method. We hope these findings inspire further exploration of perfect alignment and its applications in representation learning.