TaLK: Text-attributed Graph Dataset Distillation via Coupling Language Model with Graph-Aware Kernel
This work addresses the computational bottleneck of training LM-GNN models on large text-attributed graphs, offering a scalable distillation approach for graph learning practitioners.
TaLK introduces a dataset distillation method for text-attributed graphs that couples a language model with a graph-aware neural tangent kernel, achieving up to 97% of full-dataset performance using only 1% synthetic data.
Text-attributed graphs (TAGs) are widely used in many real-world domains, and learning on TAGs requires jointly modeling text semantics and graph structure. A standard approach for modeling TAGs is to combine a language model (LM) and a graph neural network (GNN), but joint training is computationally expensive and difficult to scale. Dataset distillation is a promising way to reduce training costs, but existing methods are not well suited to TAGs because they are typically designed for a single modality or still require repeatedly training expensive LM-GNN models on the full dataset during distillation. To address this, we propose TaLK, an effective dataset distillation method for TAGs that couples an LM with a graph-aware neural tangent kernel.This design enables efficient dataset distillation, avoiding repeated joint training on the full dataset while reflecting both textual and structural information for effective TAG learning.Experiments on multiple TAG benchmarks show that TaLK consistently outperforms existing baselines and achieves up to 97% of full-dataset performance with only 1% synthetic data.