GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
This addresses the challenge of slow GNN training for researchers and practitioners, offering a principled normalization method that enhances optimization and generalization, though it is incremental as it builds on existing normalization techniques.
The paper tackled the problem of accelerating Graph Neural Network (GNN) training by identifying effective normalization methods, showing that InstanceNorm accelerates convergence but degrades expressiveness for regular graphs, and proposing GraphNorm with a learnable shift to achieve faster convergence and better generalization, improving performance on graph classification benchmarks.
Normalization is known to help the optimization of deep neural networks. Curiously, different architectures require specialized normalization methods. In this paper, we study what normalization is effective for Graph Neural Networks (GNNs). First, we adapt and evaluate the existing methods from other domains to GNNs. Faster convergence is achieved with InstanceNorm compared to BatchNorm and LayerNorm. We provide an explanation by showing that InstanceNorm serves as a preconditioner for GNNs, but such preconditioning effect is weaker with BatchNorm due to the heavy batch noise in graph datasets. Second, we show that the shift operation in InstanceNorm results in an expressiveness degradation of GNNs for highly regular graphs. We address this issue by proposing GraphNorm with a learnable shift. Empirically, GNNs with GraphNorm converge faster compared to GNNs using other normalization. GraphNorm also improves the generalization of GNNs, achieving better performance on graph classification benchmarks.