Is This Tracker On? A Benchmark Protocol for Dynamic Tracking
This addresses the problem of evaluating and improving point tracking methods for researchers and developers, though it is incremental as it builds on existing datasets and focuses on benchmarking.
The authors introduced ITTO, a challenging benchmark suite for evaluating point tracking methods using real-world videos with human annotations, and found that existing trackers struggle, especially with re-identification after occlusion.
We introduce ITTO, a challenging new benchmark suite for evaluating and diagnosing the capabilities and limitations of point tracking methods. Our videos are sourced from existing datasets and egocentric real-world recordings, with high-quality human annotations collected through a multi-stage pipeline. ITTO captures the motion complexity, occlusion patterns, and object diversity characteristic of real-world scenes -- factors that are largely absent in current benchmarks. We conduct a rigorous analysis of state-of-the-art tracking methods on ITTO, breaking down performance along key axes of motion complexity. Our findings reveal that existing trackers struggle with these challenges, particularly in re-identifying points after occlusion, highlighting critical failure modes. These results point to the need for new modeling approaches tailored to real-world dynamics. We envision ITTO as a foundation testbed for advancing point tracking and guiding the development of more robust tracking algorithms.