CoTracker3: Simpler and Better Point Tracking by Pseudo-Labelling Real Videos
This addresses the problem of suboptimal point tracking for video analysis by reducing reliance on synthetic data, though it is incremental in simplifying existing methods.
The authors tackled the performance gap in point tracking caused by training on synthetic data by introducing CoTracker3, a simpler model trained with a semi-supervised recipe using pseudo-labels from real videos, which achieved better results with 1,000 times less data.
Most state-of-the-art point trackers are trained on synthetic data due to the difficulty of annotating real videos for this task. However, this can result in suboptimal performance due to the statistical gap between synthetic and real videos. In order to understand these issues better, we introduce CoTracker3, comprising a new tracking model and a new semi-supervised training recipe. This allows real videos without annotations to be used during training by generating pseudo-labels using off-the-shelf teachers. The new model eliminates or simplifies components from previous trackers, resulting in a simpler and often smaller architecture. This training scheme is much simpler than prior work and achieves better results using 1,000 times less data. We further study the scaling behaviour to understand the impact of using more real unsupervised data in point tracking. The model is available in online and offline variants and reliably tracks visible and occluded points.