Fast Training of Sparse Graph Neural Networks on Dense Hardware
This work addresses the challenge of scaling sparse models for researchers and practitioners in machine learning, though it is incremental as it builds on existing optimization techniques.
The paper tackled the problem of efficiently training sparse graph neural networks on dense hardware, achieving a training time of 13 minutes on a 512-core TPUv2 Pod compared to the original nearly a day.
Graph neural networks have become increasingly popular in recent years due to their ability to naturally encode relational input data and their ability to scale to large graphs by operating on a sparse representation of graph adjacency matrices. As we look to scale up these models using custom hardware, a natural assumption would be that we need hardware tailored to sparse operations and/or dynamic control flow. In this work, we question this assumption by scaling up sparse graph neural networks using a platform targeted at dense computation on fixed-size data. Drawing inspiration from optimization of numerical algorithms on sparse matrices, we develop techniques that enable training the sparse graph neural network model from Allamanis et al. [2018] in 13 minutes using a 512-core TPUv2 Pod, whereas the original training takes almost a day.