Multimodal Analogical Reasoning over Knowledge Graphs
This work addresses the need for multimodal reasoning in AI, offering a new task and dataset that could benefit fields like cognitive science and machine learning, though it appears incremental in extending analogical reasoning to multimodal contexts.
The paper tackles the problem of analogical reasoning by introducing a multimodal approach over knowledge graphs, constructing the MARS dataset and MarKG knowledge graph, and proposing the MarT framework which achieves better performance than baseline methods.
Analogical reasoning is fundamental to human cognition and holds an important place in various fields. However, previous studies mainly focus on single-modal analogical reasoning and ignore taking advantage of structure knowledge. Notably, the research in cognitive psychology has demonstrated that information from multimodal sources always brings more powerful cognitive transfer than single modality sources. To this end, we introduce the new task of multimodal analogical reasoning over knowledge graphs, which requires multimodal reasoning ability with the help of background knowledge. Specifically, we construct a Multimodal Analogical Reasoning dataSet (MARS) and a multimodal knowledge graph MarKG. We evaluate with multimodal knowledge graph embedding and pre-trained Transformer baselines, illustrating the potential challenges of the proposed task. We further propose a novel model-agnostic Multimodal analogical reasoning framework with Transformer (MarT) motivated by the structure mapping theory, which can obtain better performance. Code and datasets are available in https://github.com/zjunlp/MKG_Analogy.