Are Large Language Models In-Context Graph Learners?
This addresses the challenge of enabling LLMs to handle non-Euclidean graph structures for applications in graph learning, though it is incremental as it builds on existing RAG methods.
The paper tackles the problem of large language models (LLMs) struggling with structured graph data by conceptualizing graph learning as a retrieval-augmented generation (RAG) process, and it shows that proposed RAG frameworks significantly improve LLM performance on graph-based tasks, especially without fine-tuning or API access.
Large language models (LLMs) have demonstrated remarkable in-context reasoning capabilities across a wide range of tasks, particularly with unstructured inputs such as language or images. However, LLMs struggle to handle structured data, such as graphs, due to their lack of understanding of non-Euclidean structures. As a result, without additional fine-tuning, their performance significantly lags behind that of graph neural networks (GNNs) in graph learning tasks. In this paper, we show that learning on graph data can be conceptualized as a retrieval-augmented generation (RAG) process, where specific instances (e.g., nodes or edges) act as queries, and the graph itself serves as the retrieved context. Building on this insight, we propose a series of RAG frameworks to enhance the in-context learning capabilities of LLMs for graph learning tasks. Comprehensive evaluations demonstrate that our proposed RAG frameworks significantly improve LLM performance on graph-based tasks, particularly in scenarios where a pretrained LLM must be used without modification or accessed via an API.