Distributed Training of Graph Convolutional Networks using Subgraph Approximation
This work is significant for researchers and practitioners working with GCNs on very large graphs, as it enables distributed training without sacrificing accuracy, an incremental improvement over existing distributed graph training methods.
This paper addresses the challenge of training Graph Convolutional Networks (GCNs) on large graphs that exceed single-machine memory by proposing a distributed training strategy. Their method, which uses a subgraph approximation scheme to mitigate information loss at graph partitioning boundaries, achieves single-machine accuracy while maintaining a low memory footprint and minimizing synchronization overhead.
Modern machine learning techniques are successfully being adapted to data modeled as graphs. However, many real-world graphs are typically very large and do not fit in memory, often making the problem of training machine learning models on them intractable. Distributed training has been successfully employed to alleviate memory problems and speed up training in machine learning domains in which the input data is assumed to be independently identical distributed (i.i.d). However, distributing the training of non i.i.d data such as graphs that are used as training inputs in Graph Convolutional Networks (GCNs) causes accuracy problems since information is lost at the graph partitioning boundaries. In this paper, we propose a training strategy that mitigates the lost information across multiple partitions of a graph through a subgraph approximation scheme. Our proposed approach augments each sub-graph with a small amount of edge and vertex information that is approximated from all other sub-graphs. The subgraph approximation approach helps the distributed training system converge at single-machine accuracy, while keeping the memory footprint low and minimizing synchronization overhead between the machines.