Coarse-Fine Networks for Temporal Activity Detection in Videos
This work addresses the challenge of improving video representations for long-term motion in temporal activity localization, offering a novel approach that enhances efficiency and performance in video analysis.
The paper tackles the problem of temporal activity detection in videos by introducing Coarse-Fine Networks, a two-stream architecture that processes multiple temporal resolutions dynamically, resulting in outperforming state-of-the-art methods on datasets like Charades with reduced compute and memory usage.
In this paper, we introduce Coarse-Fine Networks, a two-stream architecture which benefits from different abstractions of temporal resolution to learn better video representations for long-term motion. Traditional Video models process inputs at one (or few) fixed temporal resolution without any dynamic frame selection. However, we argue that, processing multiple temporal resolutions of the input and doing so dynamically by learning to estimate the importance of each frame can largely improve video representations, specially in the domain of temporal activity localization. To this end, we propose (1) Grid Pool, a learned temporal downsampling layer to extract coarse features, and, (2) Multi-stage Fusion, a spatio-temporal attention mechanism to fuse a fine-grained context with the coarse features. We show that our method outperforms the state-of-the-arts for action detection in public datasets including Charades with a significantly reduced compute and memory footprint. The code is available at https://github.com/kkahatapitiya/Coarse-Fine-Networks