Neighbor-Sampling Based Momentum Stochastic Methods for Training Graph Neural Networks
This work addresses the lack of theoretical guarantees and practical elements like adaptivity and momentum in GCN training methods, offering incremental improvements for researchers and practitioners in graph representation learning.
The authors tackled the problem of training graph convolutional networks (GCNs) efficiently by developing neighbor-sampling based Adam-type stochastic methods, which achieved superior performance over existing methods, particularly on large-scale graph datasets, as demonstrated in numerical experiments on node classification tasks.
Graph convolutional networks (GCNs) are a powerful tool for graph representation learning. Due to the recursive neighborhood aggregations employed by GCNs, efficient training methods suffer from a lack of theoretical guarantees or are missing important practical elements from modern deep learning algorithms, such as adaptivity and momentum. In this paper, we present several neighbor-sampling (NS) based Adam-type stochastic methods for solving a nonconvex GCN training problem. We utilize the control variate technique proposed by [1] to reduce the stochastic error caused by neighbor sampling. Under standard assumptions for Adam-type methods, we show that our methods enjoy the optimal convergence rate. In addition, we conduct extensive numerical experiments on node classification tasks with several benchmark datasets. The results demonstrate superior performance of our methods over classic NS-based SGD that also uses the control-variate technique, especially for large-scale graph datasets. Our code is available at https://github.com/RPI-OPT/CV-ADAM-GNN .