Communication-Efficient Sampling for Distributed Training of Graph Convolutional Networks
This work addresses the problem of scalable distributed training for large real-world graphs, which is incremental as it builds on existing neighbor sampling methods by optimizing them for distributed settings.
The paper tackles the high communication cost in distributed training of Graph Convolutional Networks (GCNs) by proposing a communication-efficient neighbor sampling method that assigns higher probabilities to local nodes, reducing remote accesses. Experiments on node classification benchmarks show it significantly cuts communication overhead with minimal accuracy loss.
Training Graph Convolutional Networks (GCNs) is expensive as it needs to aggregate data recursively from neighboring nodes. To reduce the computation overhead, previous works have proposed various neighbor sampling methods that estimate the aggregation result based on a small number of sampled neighbors. Although these methods have successfully accelerated the training, they mainly focus on the single-machine setting. As real-world graphs are large, training GCNs in distributed systems is desirable. However, we found that the existing neighbor sampling methods do not work well in a distributed setting. Specifically, a naive implementation may incur a huge amount of communication of feature vectors among different machines. To address this problem, we propose a communication-efficient neighbor sampling method in this work. Our main idea is to assign higher sampling probabilities to the local nodes so that remote nodes are accessed less frequently. We present an algorithm that determines the local sampling probabilities and makes sure our skewed neighbor sampling does not affect much the convergence of the training. Our experiments with node classification benchmarks show that our method significantly reduces the communication overhead for distributed GCN training with little accuracy loss.