Online Temporal Action Localization with Memory-Augmented Transformer
This work addresses the challenge of real-time action detection in videos for applications like surveillance and autonomous systems, representing an incremental improvement over prior online methods.
The paper tackles the problem of online temporal action localization in streaming videos by proposing a memory-augmented transformer (MATR) that leverages long-term context, resulting in outperforming existing methods on THUMOS14 and MUSES datasets, including some offline approaches.
Online temporal action localization (On-TAL) is the task of identifying multiple action instances given a streaming video. Since existing methods take as input only a video segment of fixed size per iteration, they are limited in considering long-term context and require tuning the segment size carefully. To overcome these limitations, we propose memory-augmented transformer (MATR). MATR utilizes the memory queue that selectively preserves the past segment features, allowing to leverage long-term context for inference. We also propose a novel action localization method that observes the current input segment to predict the end time of the ongoing action and accesses the memory queue to estimate the start time of the action. Our method outperformed existing methods on two datasets, THUMOS14 and MUSES, surpassing not only TAL methods in the online setting but also some offline TAL methods.