MLaGA: Multimodal Large Language and Graph Assistant
This addresses the challenge of reasoning over complex graph structures with multimodal attributes for applications in real-world scenarios, representing an incremental advancement by extending LLM capabilities to this domain.
The paper tackled the problem of analyzing multimodal graphs with diverse attribute types like texts and images, which were underexplored in existing LLM-based methods, and introduced MLaGA, achieving superior performance in graph learning tasks compared to leading baselines.
Large Language Models (LLMs) have demonstrated substantial efficacy in advancing graph-structured data analysis. Prevailing LLM-based graph methods excel in adapting LLMs to text-rich graphs, wherein node attributes are text descriptions. However, their applications to multimodal graphs--where nodes are associated with diverse attribute types, such as texts and images--remain underexplored, despite their ubiquity in real-world scenarios. To bridge the gap, we introduce the Multimodal Large Language and Graph Assistant (MLaGA), an innovative model that adeptly extends LLM capabilities to facilitate reasoning over complex graph structures and multimodal attributes. We first design a structure-aware multimodal encoder to align textual and visual attributes within a unified space through a joint graph pre-training objective. Subsequently, we implement a multimodal instruction-tuning approach to seamlessly integrate multimodal features and graph structures into the LLM through lightweight projectors. Extensive experiments across multiple datasets demonstrate the effectiveness of MLaGA compared to leading baseline methods, achieving superior performance in diverse graph learning tasks under both supervised and transfer learning scenarios.