"Trust Junk" Leads to Unjustified Support for Highly Discriminatory Predictive Models
For XAI designers and developers, this work highlights the risk of visualizations causing over-trust in unfair models, but the findings are incremental as they extend known issues of persuasive visualizations to XAI.
The paper shows that providing accurate but irrelevant data in model explanations can lead to unjustified trust in discriminatory predictive models, as demonstrated through a crowdsourced study.
The persuasive power of data visualizations can go awry: for instance, in an explainable AI (XAI) context, visualizations can produce over-trust of predictive models. In this paper, we use a crowdsourced study to show that providing accurate (but superfluous or irrelevant) data in a model explanation can, in fact, result in unjustified trust and other positive beliefs about a model, even when the model is patently discriminatory and unfair. Our results suggest that XAI designers and developers need to consider the implicit or explicit rhetorics of their work, and beware of the potential of visualizations to imbue models with unearned trust.