CEIA: CLIP-Based Event-Image Alignment for Open-World Event-Based Understanding
This addresses the data scarcity problem in open-world event-based understanding for researchers and practitioners, offering a scalable solution with incremental improvements.
The paper tackles the challenge of training large event-text models due to scarce paired data by proposing CEIA, which aligns event and image data using CLIP as a bridge, achieving zero-shot superiority in applications like object recognition and retrieval.
We present CEIA, an effective framework for open-world event-based understanding. Currently training a large event-text model still poses a huge challenge due to the shortage of paired event-text data. In response to this challenge, CEIA learns to align event and image data as an alternative instead of directly aligning event and text data. Specifically, we leverage the rich event-image datasets to learn an event embedding space aligned with the image space of CLIP through contrastive learning. In this way, event and text data are naturally aligned via using image data as a bridge. Particularly, CEIA offers two distinct advantages. First, it allows us to take full advantage of the existing event-image datasets to make up the shortage of large-scale event-text datasets. Second, leveraging more training data, it also exhibits the flexibility to boost performance, ensuring scalable capability. In highlighting the versatility of our framework, we make extensive evaluations through a diverse range of event-based multi-modal applications, such as object recognition, event-image retrieval, event-text retrieval, and domain adaptation. The outcomes demonstrate CEIA's distinct zero-shot superiority over existing methods on these applications.