Self-supervised video pretraining yields robust and more human-aligned visual representations
This work addresses the mismatch in visual representation learning for AI systems by offering a more human-like approach that could benefit computer vision applications, though it is incremental in advancing pretraining paradigms.
The authors tackled the problem of learning visual representations by proposing video pretraining as an alternative to static image pretraining, resulting in VITO, which outperforms prior methods on image and video tasks, shows greater robustness to deformations, and aligns better with human judgments.
Humans learn powerful representations of objects and scenes by observing how they evolve over time. Yet, outside of specific tasks that require explicit temporal understanding, static image pretraining remains the dominant paradigm for learning visual foundation models. We question this mismatch, and ask whether video pretraining can yield visual representations that bear the hallmarks of human perception: generalisation across tasks, robustness to perturbations, and consistency with human judgements. To that end we propose a novel procedure for curating videos, and develop a contrastive framework which learns from the complex transformations therein. This simple paradigm for distilling knowledge from videos, called VITO, yields general representations that far outperform prior video pretraining methods on image understanding tasks, and image pretraining methods on video understanding tasks. Moreover, VITO representations are significantly more robust to natural and synthetic deformations than image-, video-, and adversarially-trained ones. Finally, VITO's predictions are strongly aligned with human judgements, surpassing models that were specifically trained for that purpose. Together, these results suggest that video pretraining could be a simple way of learning unified, robust, and human-aligned representations of the visual world.