Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks
This addresses a fundamental bottleneck in graph neural networks for researchers and practitioners, offering a theoretical foundation to improve model depth and performance, though it is incremental in refining existing understanding.
The study tackles the problem of over-smoothing and feature over-correlation in deep graph neural networks by showing that node representations collapse into a low-dimensional subspace, leading to performance degradation. It proposes a sum of Kronecker products property that provably prevents these issues and demonstrates empirical shortcomings in existing models.
Our study reveals new theoretical insights into over-smoothing and feature over-correlation in graph neural networks. Specifically, we demonstrate that with increased depth, node representations become dominated by a low-dimensional subspace that depends on the aggregation function but not on the feature transformations. For all aggregation functions, the rank of the node representations collapses, resulting in over-smoothing for particular aggregation functions. Our study emphasizes the importance for future research to focus on rank collapse rather than over-smoothing. Guided by our theory, we propose a sum of Kronecker products as a beneficial property that provably prevents over-smoothing, over-correlation, and rank collapse. We empirically demonstrate the shortcomings of existing models in fitting target functions of node classification tasks.