Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization
This addresses the need for efficient video representation learning for domains like action recognition, though it appears incremental by building on existing self-supervised approaches.
The paper tackles the problem of learning motion-focused video representations by proposing a self-supervised method that uses synthetic motion trajectories (tubelets) to simulate diverse motion patterns, resulting in data efficiency with performance maintained using only 25% of pretraining videos.
We propose a self-supervised method for learning motion-focused video representations. Existing approaches minimize distances between temporally augmented videos, which maintain high spatial similarity. We instead propose to learn similarities between videos with identical local motion dynamics but an otherwise different appearance. We do so by adding synthetic motion trajectories to videos which we refer to as tubelets. By simulating different tubelet motions and applying transformations, such as scaling and rotation, we introduce motion patterns beyond what is present in the pretraining data. This allows us to learn a video representation that is remarkably data efficient: our approach maintains performance when using only 25\% of the pretraining videos. Experiments on 10 diverse downstream settings demonstrate our competitive performance and generalizability to new domains and fine-grained actions.