Video Representation Learning with Joint-Embedding Predictive Architectures
This work addresses video representation learning for machine learning applications, presenting an incremental improvement with a novel regularization technique.
The paper tackles video representation learning by introducing Video JEPA with Variance-Covariance Regularization (VJ-VCR), a self-supervised method that avoids representation collapse and outperforms a generative baseline on downstream tasks requiring understanding of video dynamics.
Video representation learning is an increasingly important topic in machine learning research. We present Video JEPA with Variance-Covariance Regularization (VJ-VCR): a joint-embedding predictive architecture for self-supervised video representation learning that employs variance and covariance regularization to avoid representation collapse. We show that hidden representations from our VJ-VCR contain abstract, high-level information about the input data. Specifically, they outperform representations obtained from a generative baseline on downstream tasks that require understanding of the underlying dynamics of moving objects in the videos. Additionally, we explore different ways to incorporate latent variables into the VJ-VCR framework that capture information about uncertainty in the future in non-deterministic settings.