Cross-media Structured Common Space for Multimedia Event Extraction
This addresses the challenge of event extraction from multimedia content for applications like news analysis, though it is incremental as it builds on existing event extraction methods by incorporating visual data.
The paper tackles the problem of extracting events and arguments from multimedia documents by introducing the MultiMedia Event Extraction (M2E2) task, with results including absolute F-score gains of up to 9.8% over uni-modal methods and 8.3% over multimedia unstructured representations, and extracting 21.4% more event mentions than text-only methods.
We introduce a new task, MultiMedia Event Extraction (M2E2), which aims to extract events and their arguments from multimedia documents. We develop the first benchmark and collect a dataset of 245 multimedia news articles with extensively annotated events and arguments. We propose a novel method, Weakly Aligned Structured Embedding (WASE), that encodes structured representations of semantic information from textual and visual data into a common embedding space. The structures are aligned across modalities by employing a weakly supervised training strategy, which enables exploiting available resources without explicit cross-media annotation. Compared to uni-modal state-of-the-art methods, our approach achieves 4.0% and 9.8% absolute F-score gains on text event argument role labeling and visual event extraction. Compared to state-of-the-art multimedia unstructured representations, we achieve 8.3% and 5.0% absolute F-score gains on multimedia event extraction and argument role labeling, respectively. By utilizing images, we extract 21.4% more event mentions than traditional text-only methods.