Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions
This work addresses the challenge of video-based human-object interaction detection for computer vision applications, but it is incremental as it builds on existing weakly supervised methods with specific adaptations.
The paper tackles the problem of detecting human-object interactions in videos using only weak supervision, where the system lacks knowledge of interaction types and spatiotemporal locations, by introducing a contrastive training loss and a dataset of over 6.5k videos, resulting in improved performance over adapted baselines.
We introduce the task of weakly supervised learning for detecting human and object interactions in videos. Our task poses unique challenges as a system does not know what types of human-object interactions are present in a video or the actual spatiotemporal location of the human and the object. To address these challenges, we introduce a contrastive weakly supervised training loss that aims to jointly associate spatiotemporal regions in a video with an action and object vocabulary and encourage temporal continuity of the visual appearance of moving objects as a form of self-supervision. To train our model, we introduce a dataset comprising over 6.5k videos with human-object interaction annotations that have been semi-automatically curated from sentence captions associated with the videos. We demonstrate improved performance over weakly supervised baselines adapted to our task on our video dataset.