AllTracker: Efficient Dense Point Tracking at High Resolution
This addresses the need for efficient, high-resolution point tracking in video analysis, though it is incremental as it blends existing techniques from optical flow and point tracking.
The paper tackles the problem of long-range, dense point tracking in videos by introducing AllTracker, which estimates flow fields between a query frame and hundreds of subsequent frames, achieving state-of-the-art accuracy at high resolution (768x1024 pixels) with 16 million parameters on a 40G GPU.
We introduce AllTracker: a model that estimates long-range point tracks by way of estimating the flow field between a query frame and every other frame of a video. Unlike existing point tracking methods, our approach delivers high-resolution and dense (all-pixel) correspondence fields, which can be visualized as flow maps. Unlike existing optical flow methods, our approach corresponds one frame to hundreds of subsequent frames, rather than just the next frame. We develop a new architecture for this task, blending techniques from existing work in optical flow and point tracking: the model performs iterative inference on low-resolution grids of correspondence estimates, propagating information spatially via 2D convolution layers, and propagating information temporally via pixel-aligned attention layers. The model is fast and parameter-efficient (16 million parameters), and delivers state-of-the-art point tracking accuracy at high resolution (i.e., tracking 768x1024 pixels, on a 40G GPU). A benefit of our design is that we can train jointly on optical flow datasets and point tracking datasets, and we find that doing so is crucial for top performance. We provide an extensive ablation study on our architecture details and training recipe, making it clear which details matter most. Our code and model weights are available at https://alltracker.github.io