EgoPoints: Advancing Point Tracking for Egocentric Videos
This work addresses the challenge of tracking points in egocentric videos, which is important for applications like augmented reality and robotics, but it is incremental as it builds on existing methods with new data and metrics.
The authors tackled the problem of point tracking in egocentric videos by introducing the EgoPoints benchmark with annotated tracks and semi-real sequences, resulting in improvements such as a 2.7 percentage point increase in tracking accuracy for CoTracker and a 2.8 point gain in re-identification accuracy for PIPs++.
We introduce EgoPoints, a benchmark for point tracking in egocentric videos. We annotate 4.7K challenging tracks in egocentric sequences. Compared to the popular TAP-Vid-DAVIS evaluation benchmark, we include 9x more points that go out-of-view and 59x more points that require re-identification (ReID) after returning to view. To measure the performance of models on these challenging points, we introduce evaluation metrics that specifically monitor tracking performance on points in-view, out-of-view, and points that require re-identification. We then propose a pipeline to create semi-real sequences, with automatic ground truth. We generate 11K such sequences by combining dynamic Kubric objects with scene points from EPIC Fields. When fine-tuning point tracking methods on these sequences and evaluating on our annotated EgoPoints sequences, we improve CoTracker across all metrics, including the tracking accuracy $δ^\star_{\text{avg}}$ by 2.7 percentage points and accuracy on ReID sequences (ReID$δ_{\text{avg}}$) by 2.4 points. We also improve $δ^\star_{\text{avg}}$ and ReID$δ_{\text{avg}}$ of PIPs++ by 0.3 and 2.8 respectively.