AV-JEPA: Extending LeJEPA to Audio-Visual Self-Supervised Learning
Provides a clean, decoder-free architecture for audio-visual representation learning, but the performance is competitive rather than state-of-the-art, making it an incremental contribution.
AV-JEPA extends LeJEPA to audio-visual self-supervised learning using early-fusion Vision Transformer and modality dropout, achieving competitive classification on VGGSound (57.1% top-1) and AudioSet (32.7 mAP) and enabling zero-shot audio-video retrieval.
We present AV-JEPA, an elegant multimodal extension of LeJEPA to audio-visual self-supervised learning. Using an early-fusion Vision Transformer and modality dropout as masking, the model is trained to align the embeddings of global and per-modality local views, while the SIGReg objective encourages a theoretically optimal distribution. This achieves cross-modal alignment in the latent space, resulting in a remarkably clean architecture with no decoder, EMA teacher, complex multi-term losses, or contrastive negatives. The proposed AV-JEPA backbone delivers competitive classification performance on VGGSound (57.1% top-1) and AudioSet (32.7 mAP) and supports zero-shot audio-video retrieval out of the box.