PNL: Efficient Long-Range Dependencies Extraction with Pyramid Non-Local Module for Action Recognition
This addresses efficiency and feature extraction challenges in video action recognition, though it is incremental as it builds on existing non-local methods.
The paper tackles the high computational cost and limited regional correlation modeling of non-local blocks in action recognition by proposing a Pyramid Non-Local (PNL) module that incorporates multi-scale regional correlations. It achieves state-of-the-art performance of 83.09% on the Mini-Kinetics dataset while reducing computation cost.
Long-range spatiotemporal dependencies capturing plays an essential role in improving video features for action recognition. The non-local block inspired by the non-local means is designed to address this challenge and have shown excellent performance. However, the non-local block brings significant increase in computation cost to the original network. It also lacks the ability to model regional correlation in videos. To address the above limitations, we propose Pyramid Non-Local (PNL) module, which extends the non-local block by incorporating regional correlation at multiple scales through a pyramid structured module. This extension upscales the effectiveness of non-local operation by attending to the interaction between different regions. Empirical results prove the effectiveness and efficiency of our PNL module, which achieves state-of-the-art performance of 83.09% on the Mini-Kinetics dataset, with decreased computation cost compared to the non-local block.