Tragedy Plus Time: Capturing Unintended Human Activities from Weakly-labeled Videos
This work addresses a specific challenge in computer vision for understanding human goal-directed behavior from weakly-labeled videos, which is incremental as it builds on existing datasets and methods.
The paper tackles the problem of localizing goal-directed and unintentional human activities in videos using only video-level labels, proposing a weakly supervised algorithm with an attention mechanism and overlap regularization, and validates it on the W-Oops dataset of 2,100 videos, showing improved performance in localization and video captioning tasks.
In videos that contain actions performed unintentionally, agents do not achieve their desired goals. In such videos, it is challenging for computer vision systems to understand high-level concepts such as goal-directed behavior, an ability present in humans from a very early age. Inculcating this ability in artificially intelligent agents would make them better social learners by allowing them to evaluate human action under a teleological lens. To validate the ability of deep learning models to perform this task, we curate the W-Oops dataset, built upon the Oops dataset [15]. W-Oops consists of 2,100 unintentional human action videos, with 44 goal-directed and 30 unintentional video-level activity labels collected through human annotations. Due to the expensive segment annotation procedure, we propose a weakly supervised algorithm for localizing the goal-directed as well as unintentional temporal regions in the video leveraging solely video-level labels. In particular, we employ an attention mechanism-based strategy that predicts the temporal regions which contribute the most to a classification task. Meanwhile, our designed overlap regularization allows the model to focus on distinct portions of the video for inferring the goal-directed and unintentional activity while guaranteeing their temporal ordering. Extensive quantitative experiments verify the validity of our localization method. We further conduct a video captioning experiment which demonstrates that the proposed localization module does indeed assist teleological action understanding.