On the Limit of Explaining Black-box Temporal Graph Neural Networks
This is an incremental study that highlights fundamental constraints in interpreting black-box TGNNs, which is crucial for researchers and practitioners in graph-based machine learning.
This work investigates the limitations of perturbation-based explanation methods for Temporal Graph Neural Networks (TGNNs), showing that node-perturbation fails to reliably identify prediction paths, edge-perturbation cannot determine all contributing nodes, and combined perturbations do not reliably reveal temporal aggregation components.
Temporal Graph Neural Network (TGNN) has been receiving a lot of attention recently due to its capability in modeling time-evolving graph-related tasks. Similar to Graph Neural Networks, it is also non-trivial to interpret predictions made by a TGNN due to its black-box nature. A major approach tackling this problems in GNNs is by analyzing the model' responses on some perturbations of the model's inputs, called perturbation-based explanation methods. While these methods are convenient and flexible since they do not need internal access to the model, does this lack of internal access prevent them from revealing some important information of the predictions? Motivated by that question, this work studies the limit of some classes of perturbation-based explanation methods. Particularly, by constructing some specific instances of TGNNs, we show (i) node-perturbation cannot reliably identify the paths carrying out the prediction, (ii) edge-perturbation is not reliable in determining all nodes contributing to the prediction and (iii) perturbing both nodes and edges does not reliably help us identify the graph's components carrying out the temporal aggregation in TGNNs.