Towards Uniformity and Alignment for Multimodal Representation Learning
This work addresses a foundational problem in multimodal AI by improving representation learning for tasks like retrieval and generation, though it is incremental in refining existing InfoNCE-based approaches.
The paper tackles inherent conflicts in multimodal representation learning, such as alignment-uniformity and intra-alignment conflicts, by proposing a decoupling method that reduces distribution gaps across modalities, achieving consistent gains in retrieval and generation tasks.
Multimodal representation learning aims to construct a shared embedding space in which heterogeneous modalities are semantically aligned. Despite strong empirical results, InfoNCE-based objectives introduce inherent conflicts that yield distribution gaps across modalities. In this work, we identify two conflicts in the multimodal regime, both exacerbated as the number of modalities increases: (i) an alignment-uniformity conflict, whereby the repulsion of uniformity undermines pairwise alignment, and (ii) an intra-alignment conflict, where aligning multiple modalities induces competing alignment directions. To address these issues, we propose a principled decoupling of alignment and uniformity for multimodal representations, providing a conflict-free recipe for multimodal learning that simultaneously supports discriminative and generative use cases without task-specific modules. We then provide a theoretical guarantee that our method acts as an efficient proxy for a global Hölder divergence over multiple modality distributions, and thus reduces the distribution gap among modalities. Extensive experiments on retrieval and UnCLIP-style generation demonstrate consistent gains.