Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration
This work addresses the reliability of model explanations for practitioners in interpretable AI, though it is incremental as it builds on existing calibration techniques.
The paper tackles the problem of unreliable perturbation-based explanations due to model miscalibration under explainability-specific perturbations, and introduces ReCalX, which recalibrates models to improve explanation robustness and feature identification while preserving original predictions.
Perturbation-based explanations are widely utilized to enhance the transparency of machine-learning models in practice. However, their reliability is often compromised by the unknown model behavior under the specific perturbations used. This paper investigates the relationship between uncertainty calibration - the alignment of model confidence with actual accuracy - and perturbation-based explanations. We show that models systematically produce unreliable probability estimates when subjected to explainability-specific perturbations and theoretically prove that this directly undermines global and local explanation quality. To address this, we introduce ReCalX, a novel approach to recalibrate models for improved explanations while preserving their original predictions. Empirical evaluations across diverse models and datasets demonstrate that ReCalX consistently reduces perturbation-specific miscalibration most effectively while enhancing explanation robustness and the identification of globally important input features.