On the Prediction Instability of Graph Neural Networks
This addresses reproducibility and reliability issues in GNNs for researchers and practitioners, though it is incremental as it focuses on quantifying an existing problem rather than proposing a new solution.
The paper systematically assesses prediction instability in Graph Neural Networks (GNNs) for node classification, finding that while aggregated performance is consistent, up to one-third of incorrectly classified nodes differ across runs, with instability correlated to hyperparameters, node properties, and training set size.
Instability of trained models, i.e., the dependence of individual node predictions on random factors, can affect reproducibility, reliability, and trust in machine learning systems. In this paper, we systematically assess the prediction instability of node classification with state-of-the-art Graph Neural Networks (GNNs). With our experiments, we establish that multiple instantiations of popular GNN models trained on the same data with the same model hyperparameters result in almost identical aggregated performance but display substantial disagreement in the predictions for individual nodes. We find that up to one third of the incorrectly classified nodes differ across algorithm runs. We identify correlations between hyperparameters, node properties, and the size of the training set with the stability of predictions. In general, maximizing model performance implicitly also reduces model instability.