Leveraging Motion Information for Better Self-Supervised Video Correspondence Learning
This work addresses the problem of false matches in self-supervised video correspondence for applications such as video object segmentation and tracking, representing an incremental improvement over existing methods.
The paper tackles the challenge of achieving reliable pixel matching in self-supervised video correspondence learning by introducing a framework with a Motion Enhancement Engine and Multi-Cluster Sampler, which outperforms state-of-the-art methods on tasks like video object segmentation and keypoint tracking.
Self-supervised video correspondence learning depends on the ability to accurately associate pixels between video frames that correspond to the same visual object. However, achieving reliable pixel matching without supervision remains a major challenge. To address this issue, recent research has focused on feature learning techniques that aim to encode unique pixel representations for matching. Despite these advances, existing methods still struggle to achieve exact pixel correspondences and often suffer from false matches, limiting their effectiveness in self-supervised settings. To this end, we explore an efficient self-supervised Video Correspondence Learning framework (MER) that aims to accurately extract object details from unlabeled videos. First, we design a dedicated Motion Enhancement Engine that emphasizes capturing the dynamic motion of objects in videos. In addition, we introduce a flexible sampling strategy for inter-pixel correspondence information (Multi-Cluster Sampler) that enables the model to pay more attention to the pixel changes of important objects in motion. Through experiments, our algorithm outperforms the state-of-the-art competitors on video correspondence learning tasks such as video object segmentation and video object keypoint tracking.