Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning
This work addresses a specific bottleneck in self-supervised video correspondence learning, offering an incremental but effective solution for tasks like pose and object tracking.
The paper tackled the problem of model collapse in fully convolutional cycle-consistency training for video correspondence learning by identifying and breaking absolute position shortcuts, resulting in large improvements over vanilla methods and competitive performance in pose tracking, face landmark tracking, and video object segmentation benchmarks.
Previous cycle-consistency correspondence learning methods usually leverage image patches for training. In this paper, we present a fully convolutional method, which is simpler and more coherent to the inference process. While directly applying fully convolutional training results in model collapse, we study the underline reason behind this collapse phenomenon, indicating that the absolute positions of pixels provide a shortcut to easily accomplish cycle-consistence, which hinders the learning of meaningful visual representations. To break this absolute position shortcut, we propose to apply different crops for forward and backward frames, and adopt feature warping to establish correspondence between two crops of a same frame. The former technique enforces the corresponding pixels at forward and back tracks to have different absolute positions, and the latter effectively blocks the shortcuts going between forward and back tracks. In three label propagation benchmarks for pose tracking, face landmark tracking and video object segmentation, our method largely improves the results of vanilla fully convolutional cycle-consistency method, achieving very competitive performance compared with the self-supervised state-of-the-art approaches. Our trained model and code are available at \url{https://github.com/Steve-Tod/STFC3}.