DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap
This addresses cross-modal understanding and generation for AI applications, but it is incremental as it builds on existing contrastive methods with a new module.
The paper tackled the problem of bidirectional interactions and inherent noises in cross-modal models like CLAP and CAVP, which impact integration quality, by introducing DiffGAP, a lightweight diffusion module in contrastive space that improved performance on VGGSound and AudioCaps datasets for generation and retrieval tasks.
Recent works in cross-modal understanding and generation, notably through models like CLAP (Contrastive Language-Audio Pretraining) and CAVP (Contrastive Audio-Visual Pretraining), have significantly enhanced the alignment of text, video, and audio embeddings via a single contrastive loss. However, these methods often overlook the bidirectional interactions and inherent noises present in each modality, which can crucially impact the quality and efficacy of cross-modal integration. To address this limitation, we introduce DiffGAP, a novel approach incorporating a lightweight generative module within the contrastive space. Specifically, our DiffGAP employs a bidirectional diffusion process tailored to bridge the cross-modal gap more effectively. This involves a denoising process on text and video embeddings conditioned on audio embeddings and vice versa, thus facilitating a more nuanced and robust cross-modal interaction. Our experimental results on VGGSound and AudioCaps datasets demonstrate that DiffGAP significantly improves performance in video/text-audio generation and retrieval tasks, confirming its effectiveness in enhancing cross-modal understanding and generation capabilities.