StartNet: Online Detection of Action Start in Untrimmed Videos
This addresses the challenge of detecting action starts in real-time video streams for applications like surveillance or video analysis, representing a strong incremental advance over prior methods.
The paper tackles the problem of online detection of action starts in untrimmed, streaming videos by proposing StartNet, which decomposes the task into classification and localization stages, resulting in a 15%-30% p-mAP improvement on THUMOS'14 and comparable performance on ActivityNet with a smaller time offset.
We propose StartNet to address Online Detection of Action Start (ODAS) where action starts and their associated categories are detected in untrimmed, streaming videos. Previous methods aim to localize action starts by learning feature representations that can directly separate the start point from its preceding background. It is challenging due to the subtle appearance difference near the action starts and the lack of training data. Instead, StartNet decomposes ODAS into two stages: action classification (using ClsNet) and start point localization (using LocNet). ClsNet focuses on per-frame labeling and predicts action score distributions online. Based on the predicted action scores of the past and current frames, LocNet conducts class-agnostic start detection by optimizing long-term localization rewards using policy gradient methods. The proposed framework is validated on two large-scale datasets, THUMOS'14 and ActivityNet. The experimental results show that StartNet significantly outperforms the state-of-the-art by 15%-30% p-mAP under the offset tolerance of 1-10 seconds on THUMOS'14, and achieves comparable performance on ActivityNet with 10 times smaller time offset.