Taming the Untamed: Graph-Based Knowledge Retrieval and Reasoning for MLLMs to Conquer the Unknown
This work addresses the limitation of MLLMs in domain-specific tasks for applications like game AI, though it is incremental as it builds on existing knowledge graph and retrieval methods.
The paper tackled the problem of multimodal large language models (MLLMs) failing in rarely encountered domain-specific tasks by constructing a multimodal knowledge graph (MH-MMKG) for visual game cognition and proposing a multi-agent retriever to enhance knowledge retrieval and reasoning. Experimental results showed that the approach significantly enhanced MLLM performance, providing a new perspective on multimodal knowledge-augmented reasoning.
The real value of knowledge lies not just in its accumulation, but in its potential to be harnessed effectively to conquer the unknown. Although recent multimodal large language models (MLLMs) exhibit impressing multimodal capabilities, they often fail in rarely encountered domain-specific tasks due to limited relevant knowledge. To explore this, we adopt visual game cognition as a testbed and select Monster Hunter: World as the target to construct a multimodal knowledge graph (MH-MMKG), which incorporates multi-modalities and intricate entity relations. We also design a series of challenging queries based on MH-MMKG to evaluate the models' ability for complex knowledge retrieval and reasoning. Furthermore, we propose a multi-agent retriever that enables a model to autonomously search relevant knowledge without additional training. Experimental results show that our approach significantly enhances the performance of MLLMs, providing a new perspective on multimodal knowledge-augmented reasoning and laying a solid foundation for future research.