Zero-Shot Event Detection by Multimodal Distributional Semantic Embedding of Videos
This addresses the problem of detecting events in videos without labeled training data for researchers and practitioners in multimedia analysis, representing a novel approach rather than an incremental improvement.
The paper tackled zero-shot event detection in videos by embedding multimodal information into a distributional semantic space, achieving state-of-the-art performance with improvements from 12.6% to 13.5% in MAP and 0.73 to 0.83 in ROC-AUC, while being an order of magnitude faster.
We propose a new zero-shot Event Detection method by Multi-modal Distributional Semantic embedding of videos. Our model embeds object and action concepts as well as other available modalities from videos into a distributional semantic space. To our knowledge, this is the first Zero-Shot event detection model that is built on top of distributional semantics and extends it in the following directions: (a) semantic embedding of multimodal information in videos (with focus on the visual modalities), (b) automatically determining relevance of concepts/attributes to a free text query, which could be useful for other applications, and (c) retrieving videos by free text event query (e.g., "changing a vehicle tire") based on their content. We embed videos into a distributional semantic space and then measure the similarity between videos and the event query in a free text form. We validated our method on the large TRECVID MED (Multimedia Event Detection) challenge. Using only the event title as a query, our method outperformed the state-of-the-art that uses big descriptions from 12.6% to 13.5% with MAP metric and 0.73 to 0.83 with ROC-AUC metric. It is also an order of magnitude faster.