Explaining Predictive Uncertainty by Looking Back at Model Explanations
This work addresses the need for users to trust model predictions by explaining uncertainty, which is incremental as it builds on existing explanation methods.
The paper tackled the problem of explaining predictive uncertainty in pre-trained language models by extracting uncertain words from existing model explanations, finding these words make negative contributions to predictions and actually explain uncertainty, with experiments showing such explanations are indispensable for helping humans understand model behavior.
Predictive uncertainty estimation of pre-trained language models is an important measure of how likely people can trust their predictions. However, little is known about what makes a model prediction uncertain. Explaining predictive uncertainty is an important complement to explaining prediction labels in helping users understand model decision making and gaining their trust on model predictions, while has been largely ignored in prior works. In this work, we propose to explain the predictive uncertainty of pre-trained language models by extracting uncertain words from existing model explanations. We find the uncertain words are those identified as making negative contributions to prediction labels, while actually explaining the predictive uncertainty. Experiments show that uncertainty explanations are indispensable to explaining models and helping humans understand model prediction behavior.