ViC-MAE: Self-Supervised Representation Learning from Images and Video with Contrastive Masked Autoencoders
This addresses the need for versatile visual models in computer vision, though it is incremental as it builds on existing methods like MAE and contrastive learning.
The paper tackles the problem of learning visual representations that generalize across images and video by proposing ViC-MAE, which combines masked autoencoders and contrastive learning. The result is state-of-the-art transfer learning performance, achieving 86% top-1 accuracy on ImageNet-1k (a 1.3% improvement) and 75.9% on Something-something-v2.
We propose ViC-MAE, a model that combines both Masked AutoEncoders (MAE) and contrastive learning. ViC-MAE is trained using a global featured obtained by pooling the local representations learned under an MAE reconstruction loss and leveraging this representation under a contrastive objective across images and video frames. We show that visual representations learned under ViC-MAE generalize well to both video and image classification tasks. Particularly, ViC-MAE obtains state-of-the-art transfer learning performance from video to images on Imagenet-1k compared to the recently proposed OmniMAE by achieving a top-1 accuracy of 86% (+1.3% absolute improvement) when trained on the same data and 87.1% (+2.4% absolute improvement) when training on extra data. At the same time ViC-MAE outperforms most other methods on video benchmarks by obtaining 75.9% top-1 accuracy on the challenging Something something-v2 video benchmark . When training on videos and images from a diverse combination of datasets, our method maintains a balanced transfer-learning performance between video and image classification benchmarks, coming only as a close second to the best supervised method.