Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection
This work addresses a key bottleneck in video pre-training for computer vision researchers, offering an incremental improvement over existing masking methods.
The paper tackles the challenge of selecting appropriate masking strategies in masked video modeling by introducing a Trajectory-Aware Adaptive Token Sampler (TATS) that models motion dynamics and integrates with masked autoencoders, achieving effective performance on action recognition benchmarks like Something-Something v2 and Kinetics-400 while enabling aggressive masking and memory efficiency.
Masked video modeling~(MVM) has emerged as a highly effective pre-training strategy for visual foundation models, whereby the model reconstructs masked spatiotemporal tokens using information from visible tokens. However, a key challenge in such approaches lies in selecting an appropriate masking strategy. Previous studies have explored predefined masking techniques, including random and tube-based masking, as well as approaches that leverage key motion priors, optical flow and semantic cues from externally pre-trained models. In this work, we introduce a novel and generalizable Trajectory-Aware Adaptive Token Sampler (TATS), which models the motion dynamics of tokens and can be seamlessly integrated into the masked autoencoder (MAE) framework to select motion-centric tokens in videos. Additionally, we propose a unified training strategy that enables joint optimization of both MAE and TATS from scratch using Proximal Policy Optimization (PPO). We show that our model allows for aggressive masking without compromising performance on the downstream task of action recognition while also ensuring that the pre-training remains memory efficient. Extensive experiments of the proposed approach across four benchmarks, including Something-Something v2, Kinetics-400, UCF101, and HMDB51, demonstrate the effectiveness, transferability, generalization, and efficiency of our work compared to other state-of-the-art methods.