Layer-wise training for self-supervised learning on graphs
This addresses efficiency and scalability issues for researchers and practitioners working with large graphs, enabling deeper models on single devices, though it is incremental as it builds on existing self-supervised learning methods.
The paper tackles the memory and computational challenges of training deep graph neural networks (GNNs) on large graphs by proposing a layer-wise self-supervised algorithm, achieving similar performance to end-to-end methods with substantially increased efficiency and avoiding oversmoothing.
End-to-end training of graph neural networks (GNN) on large graphs presents several memory and computational challenges, and limits the application to shallow architectures as depth exponentially increases the memory and space complexities. In this manuscript, we propose Layer-wise Regularized Graph Infomax, an algorithm to train GNNs layer by layer in a self-supervised manner. We decouple the feature propagation and feature transformation carried out by GNNs to learn node representations in order to derive a loss function based on the prediction of future inputs. We evaluate the algorithm in inductive large graphs and show similar performance to other end to end methods and a substantially increased efficiency, which enables the training of more sophisticated models in one single device. We also show that our algorithm avoids the oversmoothing of the representations, another common challenge of deep GNNs.