Training Multimedia Event Extraction With Generated Images and Captions
This work addresses the problem of multimedia event extraction for news reporting by leveraging synthetic data to overcome data scarcity, though it is incremental as it builds on existing generation methods and training strategies.
The paper tackles the lack of annotated multimodal training data for multimedia event extraction by proposing CAMEL, which uses generated images and captions from unimodal datasets to create synthetic multimodal data, achieving state-of-the-art performance with improvements of 4.2% F1 on event mention identification and 9.8% F1 on argument identification on the M2E2 benchmark.
Contemporary news reporting increasingly features multimedia content, motivating research on multimedia event extraction. However, the task lacks annotated multimodal training data and artificially generated training data suffer from distribution shift from real-world data. In this paper, we propose Cross-modality Augmented Multimedia Event Learning (CAMEL), which successfully utilizes artificially generated multimodal training data and achieves state-of-the-art performance. We start with two labeled unimodal datasets in text and image respectively, and generate the missing modality using off-the-shelf image generators like Stable Diffusion and image captioners like BLIP. After that, we train the network on the resultant multimodal datasets. In order to learn robust features that are effective across domains, we devise an iterative and gradual training strategy. Substantial experiments show that CAMEL surpasses state-of-the-art (SOTA) baselines on the M2E2 benchmark. On multimedia events in particular, we outperform the prior SOTA by 4.2% F1 on event mention identification and by 9.8% F1 on argument identification, which indicates that CAMEL learns synergistic representations from the two modalities. Our work demonstrates a recipe to unleash the power of synthetic training data in structured prediction.