Do Users Benefit From Interpretable Vision? A User Study, Baseline, And Dataset
This work addresses the problem of evaluating interpretability methods for users in vision tasks, providing a dataset and baseline for future studies, though it is incremental in comparing existing techniques.
The study tackled whether interpretability methods benefit users more than a simple baseline of comparing inputs and predictions, finding that the baseline outperformed concept-based explanations and counterfactual explanations performed similarly but allowed slightly more accurate attribute identification.
A variety of methods exist to explain image classification models. However, whether they provide any benefit to users over simply comparing various inputs and the model's respective predictions remains unclear. We conducted a user study (N=240) to test how such a baseline explanation technique performs against concept-based and counterfactual explanations. To this end, we contribute a synthetic dataset generator capable of biasing individual attributes and quantifying their relevance to the model. In a study, we assess if participants can identify the relevant set of attributes compared to the ground-truth. Our results show that the baseline outperformed concept-based explanations. Counterfactual explanations from an invertible neural network performed similarly as the baseline. Still, they allowed users to identify some attributes more accurately. Our results highlight the importance of measuring how well users can reason about biases of a model, rather than solely relying on technical evaluations or proxy tasks. We open-source our study and dataset so it can serve as a blue-print for future studies. For code see, https://github.com/berleon/do_users_benefit_from_interpretable_vision