Long-Term 3D Point Tracking By Cost Volume Fusion
This addresses the problem of understanding non-rigid motion in the physical world for computer vision applications, representing a novel advancement beyond prior 2D-focused work.
The paper tackles long-term 3D point tracking by proposing the first deep learning framework that generalizes without test-time fine-tuning, achieving significant performance improvements over existing methods, such as outperforming 2D tracking methods even with ground truth depth and camera pose in synthetic scenarios.
Long-term point tracking is essential to understand non-rigid motion in the physical world better. Deep learning approaches have recently been incorporated into long-term point tracking, but most prior work predominantly functions in 2D. Although these methods benefit from the well-established backbones and matching frameworks, the motions they produce do not always make sense in the 3D physical world. In this paper, we propose the first deep learning framework for long-term point tracking in 3D that generalizes to new points and videos without requiring test-time fine-tuning. Our model contains a cost volume fusion module that effectively integrates multiple past appearances and motion information via a transformer architecture, significantly enhancing overall tracking performance. In terms of 3D tracking performance, our model significantly outperforms simple scene flow chaining and previous 2D point tracking methods, even if one uses ground truth depth and camera pose to backproject 2D point tracks in a synthetic scenario.