Model Generalization on Text Attribute Graphs: Principles with Large Language Models
This work addresses the problem of enhancing LLM performance on graph learning tasks with scarce labeled data for researchers and practitioners in machine learning, though it is incremental as it builds on existing methods with specific adaptations.
The paper tackled the problem of improving generalization for large language models (LLMs) on text-attributed graphs (TAGs) by addressing challenges like limited context length and embedding misalignment, resulting in a framework that achieved an 8.10% improvement with task-conditional embeddings and an additional 1.71% gain from adaptive aggregation on 11 benchmarks.
Large language models (LLMs) have recently been introduced to graph learning, aiming to extend their zero-shot generalization success to tasks where labeled graph data is scarce. Among these applications, inference over text-attributed graphs (TAGs) presents unique challenges: existing methods struggle with LLMs' limited context length for processing large node neighborhoods and the misalignment between node embeddings and the LLM token space. To address these issues, we establish two key principles for ensuring generalization and derive the framework LLM-BP accordingly: (1) Unifying the attribute space with task-adaptive embeddings, where we leverage LLM-based encoders and task-aware prompting to enhance generalization of the text attribute embeddings; (2) Developing a generalizable graph information aggregation mechanism, for which we adopt belief propagation with LLM-estimated parameters that adapt across graphs. Evaluations on 11 real-world TAG benchmarks demonstrate that LLM-BP significantly outperforms existing approaches, achieving 8.10% improvement with task-conditional embeddings and an additional 1.71% gain from adaptive aggregation. The code and task-adaptive embeddings are publicly available.