The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations
This addresses the problem of biased explanations in safety-critical applications like healthcare, highlighting an understudied challenge for the ML community, though it is incremental in proposing partial solutions.
The study audited the fairness of post-hoc explainability methods across four real-world settings, finding significant fidelity gaps between protected subgroups, and showed that pairing with robust ML can improve fairness in some cases.
Machine learning models in safety-critical settings like healthcare are often blackboxes: they contain a large number of parameters which are not transparent to users. Post-hoc explainability methods where a simple, human-interpretable model imitates the behavior of these blackbox models are often proposed to help users trust model predictions. In this work, we audit the quality of such explanations for different protected subgroups using real data from four settings in finance, healthcare, college admissions, and the US justice system. Across two different blackbox model architectures and four popular explainability methods, we find that the approximation quality of explanation models, also known as the fidelity, differs significantly between subgroups. We also demonstrate that pairing explainability methods with recent advances in robust machine learning can improve explanation fairness in some settings. However, we highlight the importance of communicating details of non-zero fidelity gaps to users, since a single solution might not exist across all settings. Finally, we discuss the implications of unfair explanation models as a challenging and understudied problem facing the machine learning community.