GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models
This addresses the challenge of integrating graph data into LLMs for researchers and practitioners, offering a more efficient alternative to existing methods, though it is incremental in improving modality alignment.
The paper tackles the problem of adapting large language models (LLMs) to handle structural graph data by proposing GRIP, a framework that fine-tunes LLMs with lightweight LoRA parameters to internalize relational information, achieving efficient performance on graph-related tasks without needing the original graph at inference.
Large Language Models (LLMs) have demonstrated remarkable capabilities in modeling sequential textual data and generalizing across diverse tasks. However, adapting LLMs to effectively handle structural data, such as knowledge graphs or web data, remains a challenging problem. Some approaches adopt complex strategies to convert graphs into text sequences, resulting in significant token overhead and rendering them impractical for large-scale graphs. Others introduce additional modules to encode graphs into fixed-size token representations for LLMs. However, these methods typically require large-scale post-training on graph-text corpus and complex alignment procedures, yet often yield sub-optimal results due to poor modality alignment. Inspired by in-parameter knowledge injection for test-time adaptation of LLMs, we propose GRIP, a novel framework that equips LLMs with the ability to internalize complex relational information from graphs through carefully designed fine-tuning tasks. This knowledge is efficiently stored within lightweight LoRA parameters, enabling the fine-tuned LLM to perform a wide range of graph-related tasks without requiring access to the original graph at inference time. Extensive experiments across multiple benchmarks validate the effectiveness and efficiency of our approach.