Few-shot Domain-Adaptive Visually-fused Event Detection from Text
This addresses the practical inefficiency of multimodal event detection models in real-world scenarios where labeled image-text pairs are scarce and visual context is unavailable at inference.
The paper tackles the problem of data scarcity and limited visual context in event detection by introducing a domain-adaptive approach that synthesizes images from text, enabling few-shot training and inference with text-only data. It achieves up to an 11-point improvement over state-of-the-art models on the M2E2 dataset.
Incorporating auxiliary modalities such as images into event detection models has attracted increasing interest over the last few years. The complexity of natural language in describing situations has motivated researchers to leverage the related visual context to improve event detection performance. However, current approaches in this area suffer from data scarcity, where a large amount of labelled text-image pairs are required for model training. Furthermore, limited access to the visual context at inference time negatively impacts the performance of such models, which makes them practically ineffective in real-world scenarios. In this paper, we present a novel domain-adaptive visually-fused event detection approach that can be trained on a few labelled image-text paired data points. Specifically, we introduce a visual imaginator method that synthesises images from text in the absence of visual context. Moreover, the imaginator can be customised to a specific domain. In doing so, our model can leverage the capabilities of pre-trained vision-language models and can be trained in a few-shot setting. This also allows for effective inference where only single-modality data (i.e. text) is available. The experimental evaluation on the benchmark M2E2 dataset shows that our model outperforms existing state-of-the-art models, by up to 11 points.