MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks
This work addresses the resource efficiency and cost reduction for training GNNs on large graphs, offering a practical alternative to distributed training for graphs that fit in single-machine storage.
The paper tackles the problem of training Graph Neural Networks (GNNs) on large-scale graphs by proposing MariusGNN, an out-of-core pipelined training system that uses a single GPU and disk storage, achieving up to 8x faster training with similar accuracy compared to state-of-the-art multi-GPU solutions.
We study training of Graph Neural Networks (GNNs) for large-scale graphs. We revisit the premise of using distributed training for billion-scale graphs and show that for graphs that fit in main memory or the SSD of a single machine, out-of-core pipelined training with a single GPU can outperform state-of-the-art (SoTA) multi-GPU solutions. We introduce MariusGNN, the first system that utilizes the entire storage hierarchy -- including disk -- for GNN training. MariusGNN introduces a series of data organization and algorithmic contributions that 1) minimize the end-to-end time required for training and 2) ensure that models learned with disk-based training exhibit accuracy similar to those fully trained in memory. We evaluate MariusGNN against SoTA systems for learning GNN models and find that single-GPU training in MariusGNN achieves the same level of accuracy up to 8x faster than multi-GPU training in these systems, thus, introducing an order of magnitude monetary cost reduction. MariusGNN is open-sourced at www.marius-project.org.