Large-Scale Automatic Labeling of Video Events with Verbs Based on Event-Participant Interaction
This work addresses video event recognition for computer vision applications, but it is incremental as it builds on existing methods with a novel feature representation.
The paper tackles the problem of labeling short video clips with English verbs by focusing on spatiotemporal interactions between event participants, achieving over 70% accuracy on a 22-class task and over 85% on 10-class subsets using only bounding box data.
We present an approach to labeling short video clips with English verbs as event descriptions. A key distinguishing aspect of this work is that it labels videos with verbs that describe the spatiotemporal interaction between event participants, humans and objects interacting with each other, abstracting away all object-class information and fine-grained image characteristics, and relying solely on the coarse-grained motion of the event participants. We apply our approach to a large set of 22 distinct verb classes and a corpus of 2,584 videos, yielding two surprising outcomes. First, a classification accuracy of greater than 70% on a 1-out-of-22 labeling task and greater than 85% on a variety of 1-out-of-10 subsets of this labeling task is independent of the choice of which of two different time-series classifiers we employ. Second, we achieve this level of accuracy using a highly impoverished intermediate representation consisting solely of the bounding boxes of one or two event participants as a function of time. This indicates that successful event recognition depends more on the choice of appropriate features that characterize the linguistic invariants of the event classes than on the particular classifier algorithms.