Centre Stage: Centricity-based Audio-Visual Temporal Action Detection
This work addresses action detection in video analysis, offering incremental improvements by integrating audio and centricity estimation into existing one-stage anchor-free architectures.
The paper tackles the problem of temporal action detection by incorporating audio modality and introducing a centricity score to improve boundary accuracy, achieving state-of-the-art performance on the EPIC-Kitchens-100 benchmark.
Previous one-stage action detection approaches have modelled temporal dependencies using only the visual modality. In this paper, we explore different strategies to incorporate the audio modality, using multi-scale cross-attention to fuse the two modalities. We also demonstrate the correlation between the distance from the timestep to the action centre and the accuracy of the predicted boundaries. Thus, we propose a novel network head to estimate the closeness of timesteps to the action centre, which we call the centricity score. This leads to increased confidence for proposals that exhibit more precise boundaries. Our method can be integrated with other one-stage anchor-free architectures and we demonstrate this on three recent baselines on the EPIC-Kitchens-100 action detection benchmark where we achieve state-of-the-art performance. Detailed ablation studies showcase the benefits of fusing audio and our proposed centricity scores. Code and models for our proposed method are publicly available at https://github.com/hanielwang/Audio-Visual-TAD.git