Visualization and Analysis of the Loss Landscape in Graph Neural Networks
This work addresses the need for better insights into GNN training dynamics to improve efficiency and performance in graph-based applications, though it is incremental in nature.
The paper tackled the problem of understanding GNN optimization and generalization by introducing a learnable dimensionality reduction method for visualizing loss landscapes, which surpassed PCA-based approaches with lower memory usage and enabled analysis of architectural and training factors.
Graph Neural Networks (GNNs) are powerful models for graph-structured data, with broad applications. However, the interplay between GNN parameter optimization, expressivity, and generalization remains poorly understood. We address this by introducing an efficient learnable dimensionality reduction method for visualizing GNN loss landscapes, and by analyzing the effects of over-smoothing, jumping knowledge, quantization, sparsification, and preconditioner on GNN optimization. Our learnable projection method surpasses the state-of-the-art PCA-based approach, enabling accurate reconstruction of high-dimensional parameters with lower memory usage. We further show that architecture, sparsification, and optimizer's preconditioning significantly impact the GNN optimization landscape and their training process and final prediction performance. These insights contribute to developing more efficient designs of GNN architectures and training strategies.