MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion
This work improves multimodal knowledge graph completion for AI applications by reducing modality interference, though it is incremental as it builds on existing frameworks.
The paper tackles the problem of multimodal knowledge graph completion by addressing mutual interference between modalities and equal treatment of inputs, proposing MoSE which learns modality-split embeddings and uses ensemble inference, resulting in outperforming state-of-the-art methods on three datasets.
Multimodal knowledge graph completion (MKGC) aims to predict missing entities in MKGs. Previous works usually share relation representation across modalities. This results in mutual interference between modalities during training, since for a pair of entities, the relation from one modality probably contradicts that from another modality. Furthermore, making a unified prediction based on the shared relation representation treats the input in different modalities equally, while their importance to the MKGC task should be different. In this paper, we propose MoSE, a Modality Split representation learning and Ensemble inference framework for MKGC. Specifically, in the training phase, we learn modality-split relation embeddings for each modality instead of a single modality-shared one, which alleviates the modality interference. Based on these embeddings, in the inference phase, we first make modality-split predictions and then exploit various ensemble methods to combine the predictions with different weights, which models the modality importance dynamically. Experimental results on three KG datasets show that MoSE outperforms state-of-the-art MKGC methods. Codes are available at https://github.com/OreOZhao/MoSE4MKGC.