Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation
For researchers working on graph learning with language models, this work provides a novel method to encode whole-graph information via weight updates, improving zero-shot performance.
The paper addresses the challenge of adapting language models to graph tasks by proposing a weight-level information injection paradigm that generates task-specific LoRA updates conditioned on graph structure. GaRA consistently outperforms baselines on zero-shot graph learning tasks.
Graph neural networks (GNNs) tightly couple their input-output parameters to dataset-specific feature spaces and target sets, exhibiting limited transferability across different datasets. In contrast, language models (LMs) generalize flexibly via a unified input-output interface, motivating recent attempts to adapt LMs to graph tasks. However, existing methods struggle to encode whole-graph information, leading to potential information loss and suboptimal graph understanding. In this work, we propose a novel weight-level information injection paradigm for adapting LMs to graph tasks. This paradigm injects whole-graph information by generating task-specific weight updates that interact directly with hidden representations. Instantiating this paradigm following low-rank adaptation (LoRA), we introduce GaRA, a Graph-aware LoRA generation model. GaRA constructs low-rank weight updates conditioned on the original graph structures and constrains the norm of the generated updates, thus injecting whole-graph information and avoiding the optimization bias in the weight generation. Empirical studies demonstrate that GaRA consistently outperforms baselines on zero-shot graph learning tasks.