Learning Concept Taxonomies from Multi-modal Data
This addresses the problem of building concept taxonomies from multi-modal data for AI and NLP applications, representing a novel method for a known bottleneck.
The authors tackled the problem of automatically building hypernym taxonomies by jointly leveraging textual and visual data, proposing a probabilistic model with end-to-end features based on distributed representations. Their system outperformed previous approaches by a large gap when evaluated on WordNet hierarchies.
We study the problem of automatically building hypernym taxonomies from textual and visual data. Previous works in taxonomy induction generally ignore the increasingly prominent visual data, which encode important perceptual semantics. Instead, we propose a probabilistic model for taxonomy induction by jointly leveraging text and images. To avoid hand-crafted feature engineering, we design end-to-end features based on distributed representations of images and words. The model is discriminatively trained given a small set of existing ontologies and is capable of building full taxonomies from scratch for a collection of unseen conceptual label items with associated images. We evaluate our model and features on the WordNet hierarchies, where our system outperforms previous approaches by a large gap.