Tracking by Predicting 3-D Gaussians Over Time
This addresses video understanding and tracking for computer vision applications, offering a novel self-supervised method with strong performance gains.
The paper tackles video representation learning by encoding image sequences into moving 3-D Gaussian splats, which enforces a 3-D scene consistency bias and enables tracking to emerge during pretraining. The method achieves zero-shot tracking performance comparable to state-of-the-art and, with finetuning, improves by 34.6% on Kinetics and 13.1% on Kubric datasets over existing self-supervised approaches.
We propose Video Gaussian Masked Autoencoders (Video-GMAE), a self-supervised approach for representation learning that encodes a sequence of images into a set of Gaussian splats moving over time. Representing a video as a set of Gaussians enforces a reasonable inductive bias: that 2-D videos are often consistent projections of a dynamic 3-D scene. We find that tracking emerges when pretraining a network with this architecture. Mapping the trajectory of the learnt Gaussians onto the image plane gives zero-shot tracking performance comparable to state-of-the-art. With small-scale finetuning, our models achieve 34.6% improvement on Kinetics, and 13.1% on Kubric datasets, surpassing existing self-supervised video approaches. The project page and code are publicly available at https://videogmae.org/ and https://github.com/tekotan/video-gmae.