Forget, Anticipate and Adapt: Test Time Training for Long Videos
For researchers working on video understanding and test-time adaptation, this work addresses the computational bottleneck of applying TTT to long videos, though the novelty is incremental as it builds on existing TTT frameworks.
The paper tackles test-time training for long videos, where existing methods become computationally intractable for hours-long videos. The proposed Frame Forgetting Network (FFN) operates on only three frames per sliding window and uses a surprise metric for adaptive windowing, achieving effective temporal context retention and reducing compute, with empirical results on dense segmentation, video classification, depth estimation, and multi-hour videos.
Test Time Training (TTT) is a mechanism in which a model adapts to an incoming test-sample by performing some self-supervised (SSL) task and updating its weights even during inference. This procedure does not require labels at test-time. This paper focuses on TTT for long-videos. A major concern with existing approaches is: 1) they perform TTT updates using a sliding window containing frames in the past, whose compute increases linearly with the size of window. This becomes computationally intractable when the videos are hours long. 2) TTT is performed even when temporally close frames look similar, thereby consuming a lot of compute. We present the Frame Forgetting Network (FFN) that: 1) operates on only three frames within the sliding window, namely the frame that exits, the current frame and the frame after that. The model still manages to retain temporal context and work for hours long-videos; 2) mathematically define a surprise metric: how much new information the incoming frame contains with respect to the past seen frame. This facilitates determining how to modify the effective window size during TTT and constitutes the core mechanism of an adaptive windowing algorithm. Additionally, we curate a dataset EpicTours containing up to 3 hour long videos of walking city-tours, whereas earlier datasets on this problem were only 5 min long. We demonstrate FFNs empirical effectiveness on dense-segmentation, video classification tasks, generalization to depth-estimation, and multi-hour long videos.