Introducing Gating and Context into Temporal Action Detection
This work addresses action detection in videos, which is important for applications like surveillance and video analysis, but it is incremental as it builds on existing transformer-based insights.
The paper tackles the challenge of temporal action detection in untrimmed videos by proposing a refined feature extraction process using a gating mechanism and context branch, resulting in consistent improvements over baselines on THUMOS14 and EPIC-KITCHEN 100 datasets.
Temporal Action Detection (TAD), the task of localizing and classifying actions in untrimmed video, remains challenging due to action overlaps and variable action durations. Recent findings suggest that TAD performance is dependent on the structural design of transformers rather than on the self-attention mechanism. Building on this insight, we propose a refined feature extraction process through lightweight, yet effective operations. First, we employ a local branch that employs parallel convolutions with varying window sizes to capture both fine-grained and coarse-grained temporal features. This branch incorporates a gating mechanism to select the most relevant features. Second, we introduce a context branch that uses boundary frames as key-value pairs to analyze their relationship with the central frame through cross-attention. The proposed method captures temporal dependencies and improves contextual understanding. Evaluations of the gating mechanism and context branch on challenging datasets (THUMOS14 and EPIC-KITCHEN 100) show a consistent improvement over the baseline and existing methods.