AlignMamba: Enhancing Multimodal Mamba with Local and Global Cross-modal Alignment
This work addresses the problem of computational inefficiency in multimodal alignment for researchers and practitioners, offering an incremental improvement over existing Mamba-based methods.
The paper tackled the challenge of efficiently modeling cross-modal relationships in multimodal fusion by proposing AlignMamba, which integrates local and global alignment modules with a Mamba backbone, achieving improved performance on tasks like complete and incomplete multimodal fusion.
Cross-modal alignment is crucial for multimodal representation fusion due to the inherent heterogeneity between modalities. While Transformer-based methods have shown promising results in modeling inter-modal relationships, their quadratic computational complexity limits their applicability to long-sequence or large-scale data. Although recent Mamba-based approaches achieve linear complexity, their sequential scanning mechanism poses fundamental challenges in comprehensively modeling cross-modal relationships. To address this limitation, we propose AlignMamba, an efficient and effective method for multimodal fusion. Specifically, grounded in Optimal Transport, we introduce a local cross-modal alignment module that explicitly learns token-level correspondences between different modalities. Moreover, we propose a global cross-modal alignment loss based on Maximum Mean Discrepancy to implicitly enforce the consistency between different modal distributions. Finally, the unimodal representations after local and global alignment are passed to the Mamba backbone for further cross-modal interaction and multimodal fusion. Extensive experiments on complete and incomplete multimodal fusion tasks demonstrate the effectiveness and efficiency of the proposed method.