Self-supervised visual learning from interactions with objects
This work addresses the robustness gap in self-supervised learning for computer vision by leveraging embodied interactions, offering an incremental improvement for domain-specific applications like object recognition.
The paper tackled the problem of self-supervised visual learning lacking robustness compared to human vision by incorporating object-related actions from video datasets, resulting in consistent outperformance of previous methods on downstream category recognition with improved viewpoint-wise alignment.
Self-supervised learning (SSL) has revolutionized visual representation learning, but has not achieved the robustness of human vision. A reason for this could be that SSL does not leverage all the data available to humans during learning. When learning about an object, humans often purposefully turn or move around objects and research suggests that these interactions can substantially enhance their learning. Here we explore whether such object-related actions can boost SSL. For this, we extract the actions performed to change from one ego-centric view of an object to another in four video datasets. We then introduce a new loss function to learn visual and action embeddings by aligning the performed action with the representations of two images extracted from the same clip. This permits the performed actions to structure the latent visual representation. Our experiments show that our method consistently outperforms previous methods on downstream category recognition. In our analysis, we find that the observed improvement is associated with a better viewpoint-wise alignment of different objects from the same category. Overall, our work demonstrates that embodied interactions with objects can improve SSL of object categories.