ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos
This addresses the challenge of accurately detecting dynamic interactions in videos for computer vision applications, representing an incremental improvement by extending HOI detection to the temporal domain.
The paper tackles the problem of detecting temporal human-object interactions in videos, where existing static image methods fail, by proposing the ST-HOI architecture that uses temporal information like trajectories and features, achieving strong performance as a baseline on the new VidHOI benchmark.
Detecting human-object interactions (HOI) is an important step toward a comprehensive visual understanding of machines. While detecting non-temporal HOIs (e.g., sitting on a chair) from static images is feasible, it is unlikely even for humans to guess temporal-related HOIs (e.g., opening/closing a door) from a single video frame, where the neighboring frames play an essential role. However, conventional HOI methods operating on only static images have been used to predict temporal-related interactions, which is essentially guessing without temporal contexts and may lead to sub-optimal performance. In this paper, we bridge this gap by detecting video-based HOIs with explicit temporal information. We first show that a naive temporal-aware variant of a common action detection baseline does not work on video-based HOIs due to a feature-inconsistency issue. We then propose a simple yet effective architecture named Spatial-Temporal HOI Detection (ST-HOI) utilizing temporal information such as human and object trajectories, correctly-localized visual features, and spatial-temporal masking pose features. We construct a new video HOI benchmark dubbed VidHOI where our proposed approach serves as a solid baseline.