MJEPA: A Simple and Scalable Joint-Embedding Predictive Architecture for Audio-Visual Learning
This work provides a simple and scalable method for self-supervised audio-visual learning, benefiting researchers in multimodal representation learning by demonstrating that cross-modal prediction with a shared encoder can improve both modalities.
MJEPA introduces a simple, scalable joint-embedding predictive architecture for audio-visual learning using a single unified encoder and a single predictive objective. It achieves state-of-the-art results, outperforming prior frozen baselines by over 6.8 mAP on AudioSet-20K and surpassing fully finetuned models on ESC-50 and FSD50K, while using 10x less video data.
Self-supervised learning from large-scale video data has emerged as a dominant paradigm for visual representation learning. Since audio and visual streams naturally co-occur in video data, extending this success to jointly learn from both modalities is a natural next step, yet it remains challenging. Existing audio-visual self-supervised methods rely on modality-specific encoders and complex combinations of contrastive or reconstruction objectives, limiting cross-modal synergy and scalability. Joint Embedding Predictive Architectures (JEPAs) offer a simple, modality-agnostic alternative, but have to date been applied primarily to individual modalities. We introduce MJEPA, a joint-embedding predictive architecture for audio-visual learning that uses a single, unified encoder for both modalities. Our approach uses only a single predictive objective, applied both within and across modalities. We show that cross-modal prediction is critical: without it, a shared encoder degrades below unimodal baselines; with it, each modality's representation benefits from the other. Our frozen ViT-g model outperforms the best prior frozen baseline by over 6.8 mAP on AudioSet-20K, surpasses fully finetuned models on ESC-50 and FSD50K, and is competitive on video benchmarks despite using 10x less video data.