MOVER: Multimodal Optimal Transport with Volume-based Embedding Regularization
This addresses the challenge of building semantically structured and generalizable multimodal representations for applications like retrieval, though it appears incremental as it builds on existing contrastive and optimal transport methods.
The paper tackled the problem of aligning multiple modalities like text, video, and audio in multimodal learning, which struggles with generalization and semantic structure, by proposing MOVER, a framework that combines optimal transport-based alignment with volume-based regularization, resulting in significant outperformance over prior state-of-the-art methods in retrieval tasks.
Recent advances in multimodal learning have largely relied on pairwise contrastive objectives to align different modalities, such as text, video, and audio, in a shared embedding space. While effective in bi-modal setups, these approaches struggle to generalize across multiple modalities and often lack semantic structure in high-dimensional spaces. In this paper, we propose MOVER, a novel framework that combines optimal transport-based soft alignment with volume-based geometric regularization to build semantically aligned and structured multimodal representations. By integrating a transport-guided matching mechanism with a geometric volume minimization objective (GAVE), MOVER encourages consistent alignment across all modalities in a modality-agnostic manner. Experiments on text-video-audio retrieval tasks demonstrate that MOVER significantly outperforms prior state-of-the-art methods in both zero-shot and finetuned settings. Additional analysis shows improved generalization to unseen modality combinations and stronger structural consistency in the learned embedding space.