Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks
This work addresses prediction instability in graph neural networks for node classification, offering an incremental improvement in knowledge distillation techniques.
The paper tackles prediction churn in graph neural networks by analyzing feature utilization differences between models with different initializations, proposing Influence Difference (ID) as a metric and DropDistillation (DD) to minimize it in knowledge distillation. DD outperforms previous methods in prediction stability and overall performance across six benchmark datasets for node classification.
Models with similar performances exhibit significant disagreement in the predictions of individual samples, referred to as prediction churn. Our work explores this phenomenon in graph neural networks by investigating differences between models differing only in their initializations in their utilized features for predictions. We propose a novel metric called Influence Difference (ID) to quantify the variation in reasons used by nodes across models by comparing their influence distribution. Additionally, we consider the differences between nodes with a stable and an unstable prediction, positing that both equally utilize different reasons and thus provide a meaningful gradient signal to closely match two models even when the predictions for nodes are similar. Based on our analysis, we propose to minimize this ID in Knowledge Distillation, a domain where a new model should closely match an established one. As an efficient approximation, we introduce DropDistillation (DD) that matches the output for a graph perturbed by edge deletions. Our empirical evaluation of six benchmark datasets for node classification validates the differences in utilized features. DD outperforms previous methods regarding prediction stability and overall performance in all considered Knowledge Distillation experiments.