Noise-powered Multi-modal Knowledge Graph Representation Framework
This addresses the need for unified multi-modal knowledge graph representation learning to improve multi-modal AI systems, though it appears incremental as it builds on existing Transformer architectures.
The paper tackles the problem of embedding structured knowledge into multi-modal Large Language Models to reduce knowledge misconceptions and hallucinations, proposing a Transformer-based method with modality-level noise masking that achieves state-of-the-art performance on ten datasets for multi-modal knowledge graph completion and entity alignment tasks.
The rise of Multi-modal Pre-training highlights the necessity for a unified Multi-Modal Knowledge Graph (MMKG) representation learning framework. Such a framework is essential for embedding structured knowledge into multi-modal Large Language Models effectively, alleviating issues like knowledge misconceptions and multi-modal hallucinations. In this work, we explore the efficacy of models in accurately embedding entities within MMKGs through two pivotal tasks: Multi-modal Knowledge Graph Completion (MKGC) and Multi-modal Entity Alignment (MMEA). Building on this foundation, we propose a novel SNAG method that utilizes a Transformer-based architecture equipped with modality-level noise masking to robustly integrate multi-modal entity features in KGs. By incorporating specific training objectives for both MKGC and MMEA, our approach achieves SOTA performance across a total of ten datasets, demonstrating its versatility. Moreover, SNAG can not only function as a standalone model but also enhance other existing methods, providing stable performance improvements. Code and data are available at https://github.com/zjukg/SNAG.