Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
This work addresses the problem of making MLPs more effective for graph-based tasks, offering a method that enhances performance in a specific domain, though it appears incremental by building on existing knowledge distillation approaches.
The paper tackles the challenge of effectively injecting structural information into MLPs for graph learning by proposing Propagate & Distill (P&D), which propagates teacher GNN outputs before knowledge distillation, resulting in improved performance on semi-supervised node classification tasks as demonstrated on real-world benchmark datasets.
Recent studies attempted to utilize multilayer perceptrons (MLPs) to solve semi-supervised node classification on graphs, by training a student MLP by knowledge distillation (KD) from a teacher graph neural network (GNN). While previous studies have focused mostly on training the student MLP by matching the output probability distributions between the teacher and student models during KD, it has not been systematically studied how to inject the structural information in an explicit and interpretable manner. Inspired by GNNs that separate feature transformation $T$ and propagation $Π$, we re-frame the KD process as enabling the student MLP to explicitly learn both $T$ and $Π$. Although this can be achieved by applying the inverse propagation $Π^{-1}$ before distillation from the teacher GNN, it still comes with a high computational cost from large matrix multiplications during training. To solve this problem, we propose Propagate & Distill (P&D), which propagates the output of the teacher GNN before KD and can be interpreted as an approximate process of the inverse propagation $Π^{-1}$. Through comprehensive evaluations using real-world benchmark datasets, we demonstrate the effectiveness of P&D by showing further performance boost of the student MLP.