TorchGT: A Holistic System for Large-scale Graph Transformer Training
This addresses the practical adoption bottleneck for graph transformers in real-world applications, representing a domain-specific advancement.
The paper tackles the problem of scaling graph transformers to large-scale graphs with millions of nodes by proposing TorchGT, a system that optimizes training at algorithm, runtime, and kernel levels, resulting in up to 62.7x faster training and support for graph sequences up to 1M in length.
Graph Transformer is a new architecture that surpasses GNNs in graph learning. While there emerge inspiring algorithm advancements, their practical adoption is still limited, particularly on real-world graphs involving up to millions of nodes. We observe existing graph transformers fail on large-scale graphs mainly due to heavy computation, limited scalability and inferior model quality. Motivated by these observations, we propose TorchGT, the first efficient, scalable, and accurate graph transformer training system. TorchGT optimizes training at different levels. At algorithm level, by harnessing the graph sparsity, TorchGT introduces a Dual-interleaved Attention which is computation-efficient and accuracy-maintained. At runtime level, TorchGT scales training across workers with a communication-light Cluster-aware Graph Parallelism. At kernel level, an Elastic Computation Reformation further optimizes the computation by reducing memory access latency in a dynamic way. Extensive experiments demonstrate that TorchGT boosts training by up to 62.7x and supports graph sequence lengths of up to 1M.