CoWTracker: Tracking by Warping instead of Correlation
This addresses efficiency and scalability issues in computer vision for applications such as video analysis and robotics, representing a novel approach rather than an incremental improvement.
The paper tackled the problem of dense point tracking by proposing a method that replaces cost volumes with warping, achieving state-of-the-art performance on benchmarks like TAP-Vid-DAVIS and TAP-Vid-Kinetics, and also excelling at optical flow tasks on Sintel and KITTI.
Dense point tracking is a fundamental problem in computer vision, with applications ranging from video analysis to robotic manipulation. State-of-the-art trackers typically rely on cost volumes to match features across frames, but this approach incurs quadratic complexity in spatial resolution, limiting scalability and efficiency. In this paper, we propose \method, a novel dense point tracker that eschews cost volumes in favor of warping. Inspired by recent advances in optical flow, our approach iteratively refines track estimates by warping features from the target frame to the query frame based on the current estimate. Combined with a transformer architecture that performs joint spatiotemporal reasoning across all tracks, our design establishes long-range correspondences without computing feature correlations. Our model is simple and achieves state-of-the-art performance on standard dense point tracking benchmarks, including TAP-Vid-DAVIS, TAP-Vid-Kinetics, and Robo-TAP. Remarkably, the model also excels at optical flow, sometimes outperforming specialized methods on the Sintel, KITTI, and Spring benchmarks. These results suggest that warping-based architectures can unify dense point tracking and optical flow estimation.