X Hacking: The Threat of Misguided AutoML
This addresses a credibility and reproducibility issue in explainable AI for researchers and practitioners, highlighting a novel vulnerability in automated pipelines.
The paper tackles the problem of analysts manipulating explainable AI (XAI) metrics to support pre-specified conclusions, introducing X-hacking as a form of p-hacking, and shows that Bayesian optimization accelerates this process 3-fold on average for susceptible features compared to random sampling.
Explainable AI (XAI) and interpretable machine learning methods help to build trust in model predictions and derived insights, yet also present a perverse incentive for analysts to manipulate XAI metrics to support pre-specified conclusions. This paper introduces the concept of X-hacking, a form of p-hacking applied to XAI metrics such as SHAP values. We show how easily an automated machine learning pipeline can be adapted to exploit model multiplicity at scale: searching a Rashomon set of 'defensible' models with similar predictive performance to find a desired explanation. We formulate the trade-off between explanation and accuracy as a multi-objective optimisation problem, and illustrate empirically on familiar real-world datasets that, on average, Bayesian optimisation accelerates X-hacking 3-fold for features susceptible to it, versus random sampling. We show the vulnerability of a dataset to X-hacking can be determined by information redundancy among features. Finally, we suggest possible methods for detection and prevention, and discuss ethical implications for the credibility and reproducibility of XAI.