Single-GPU GNN Systems: Traps and Pitfalls
This work critiques and aims to improve evaluation practices in GNN systems for researchers and practitioners, though it is incremental as it builds on existing systems.
The paper identifies that current single-GPU GNN systems often omit training accuracy results and rely on small datasets, leading to pitfalls in design and evaluation that question the practicality of optimizations. It proposes a new reference system with recommendations to address these issues and advance the state-of-the-art.
The current graph neural network (GNN) systems have established a clear trend of not showing training accuracy results, and directly or indirectly relying on smaller datasets for evaluations majorly. Our in-depth analysis shows that it leads to a chain of pitfalls in the system design and evaluation process, questioning the practicality of many of the proposed system optimizations, and affecting conclusions and lessons learned. We analyze many single-GPU systems and show the fundamental impact of these pitfalls. We further develop hypotheses, recommendations, and evaluation methodologies, and provide future directions. Finally, a new reference system is developed to establish a new line of optimizations rooted in solving the system-design pitfalls efficiently and practically. The proposed design can productively be integrated into prior works, thereby truly advancing the state-of-the-art.