Random Temporal Skipping for Multirate Video Analysis
This addresses video analysis for applications like action recognition, but it is incremental as it builds on existing temporal jittering methods.
The paper tackles the problem of video understanding for multirate videos where actions occur at different speeds, proposing random temporal skipping to vary frame sampling rates during training, and achieves state-of-the-art performance on six benchmarks with real-time operation.
Current state-of-the-art approaches to video understanding adopt temporal jittering to simulate analyzing the video at varying frame rates. However, this does not work well for multirate videos, in which actions or subactions occur at different speeds. The frame sampling rate should vary in accordance with the different motion speeds. In this work, we propose a simple yet effective strategy, termed random temporal skipping, to address this situation. This strategy effectively handles multirate videos by randomizing the sampling rate during training. It is an exhaustive approach, which can potentially cover all motion speed variations. Furthermore, due to the large temporal skipping, our network can see video clips that originally cover over 100 frames. Such a time range is enough to analyze most actions/events. We also introduce an occlusion-aware optical flow learning method that generates improved motion maps for human action recognition. Our framework is end-to-end trainable, runs in real-time, and achieves state-of-the-art performance on six widely adopted video benchmarks.