Event Specific Multimodal Pattern Mining with Image-Caption Pairs
This work addresses the challenge of extracting human-recognizable and named patterns from multimodal data for news event analysis, representing an incremental advancement in pattern mining techniques.
The paper tackles the problem of discovering semantically meaningful image patch patterns from weakly supervised image-caption pairs in news events, resulting in patterns that are 26.2% more semantically meaningful than state-of-the-art vision-only methods and achieving 54.5% accuracy in tagging patches without direct supervision.
In this paper we describe a novel framework and algorithms for discovering image patch patterns from a large corpus of weakly supervised image-caption pairs generated from news events. Current pattern mining techniques attempt to find patterns that are representative and discriminative, we stipulate that our discovered patterns must also be recognizable by humans and preferably with meaningful names. We propose a new multimodal pattern mining approach that leverages the descriptive captions often accompanying news images to learn semantically meaningful image patch patterns. The mutltimodal patterns are then named using words mined from the associated image captions for each pattern. A novel evaluation framework is provided that demonstrates our patterns are 26.2% more semantically meaningful than those discovered by the state of the art vision only pipeline, and that we can provide tags for the discovered images patches with 54.5% accuracy with no direct supervision. Our methods also discover named patterns beyond those covered by the existing image datasets like ImageNet. To the best of our knowledge this is the first algorithm developed to automatically mine image patch patterns that have strong semantic meaning specific to high-level news events, and then evaluate these patterns based on that criteria.