Enhanced Soups for Graph Neural Networks
This addresses scalability issues for researchers and practitioners using GNNs in scientific and HPC applications, offering incremental improvements over existing souping methods.
The paper tackles the problem of slow and memory-intensive souping algorithms for Graph Neural Networks (GNNs) by introducing Learned Souping and Partition Learned Souping, achieving up to 1.2% accuracy improvement, 2.1X speedup, and 76% memory reduction on benchmark datasets.
Graph Neural Networks (GNN) have demonstrated state-of-the-art performance in numerous scientific and high-performance computing (HPC) applications. Recent work suggests that "souping" (combining) individually trained GNNs into a single model can improve performance without increasing compute and memory costs during inference. However, existing souping algorithms are often slow and memory-intensive, which limits their scalability. We introduce Learned Souping for GNNs, a gradient-descent-based souping strategy that substantially reduces time and memory overhead compared to existing methods. Our approach is evaluated across multiple Open Graph Benchmark (OGB) datasets and GNN architectures, achieving up to 1.2% accuracy improvement and 2.1X speedup. Additionally, we propose Partition Learned Souping, a novel partition-based variant of learned souping that significantly reduces memory usage. On the ogbn-products dataset with GraphSAGE, partition learned souping achieves a 24.5X speedup and a 76% memory reduction without compromising accuracy.