Action Unit Memory Network for Weakly Supervised Temporal Action Localization
This work addresses the challenge of detecting and localizing actions in untrimmed videos with only video-level labels, which is incremental as it builds on existing methods with a novel memory-based approach.
The paper tackles the problem of weakly supervised temporal action localization in videos by proposing an Action Unit Memory Network (AUMN) to improve localization completeness and reduce background interference, resulting in an average mAP improvement from 47.0% to 52.1% on the THUMOS14 dataset.
Weakly supervised temporal action localization aims to detect and localize actions in untrimmed videos with only video-level labels during training. However, without frame-level annotations, it is challenging to achieve localization completeness and relieve background interference. In this paper, we present an Action Unit Memory Network (AUMN) for weakly supervised temporal action localization, which can mitigate the above two challenges by learning an action unit memory bank. In the proposed AUMN, two attention modules are designed to update the memory bank adaptively and learn action units specific classifiers. Furthermore, three effective mechanisms (diversity, homogeneity and sparsity) are designed to guide the updating of the memory network. To the best of our knowledge, this is the first work to explicitly model the action units with a memory network. Extensive experimental results on two standard benchmarks (THUMOS14 and ActivityNet) demonstrate that our AUMN performs favorably against state-of-the-art methods. Specifically, the average mAP of IoU thresholds from 0.1 to 0.5 on the THUMOS14 dataset is significantly improved from 47.0% to 52.1%.