On Cognitive Preferences and the Plausibility of Rule-based Models
This work questions a foundational assumption in interpretable machine learning, potentially impacting how models are designed for user acceptance, though it is incremental as it builds on existing cognitive bias research.
The paper challenges the assumption that simpler rule-based models are always more interpretable, focusing on plausibility as user acceptance, and finds through a crowd-sourcing study with about 3,000 judgments that there is no strong preference for simple rules, with a weak preference for longer rules in some domains.
It is conventional wisdom in machine learning and data mining that logical models such as rule sets are more interpretable than other models, and that among such rule-based models, simpler models are more interpretable than more complex ones. In this position paper, we question this latter assumption by focusing on one particular aspect of interpretability, namely the plausibility of models. Roughly speaking, we equate the plausibility of a model with the likeliness that a user accepts it as an explanation for a prediction. In particular, we argue that, all other things being equal, longer explanations may be more convincing than shorter ones, and that the predominant bias for shorter models, which is typically necessary for learning powerful discriminative models, may not be suitable when it comes to user acceptance of the learned models. To that end, we first recapitulate evidence for and against this postulate, and then report the results of an evaluation in a crowd-sourcing study based on about 3.000 judgments. The results do not reveal a strong preference for simple rules, whereas we can observe a weak preference for longer rules in some domains. We then relate these results to well-known cognitive biases such as the conjunction fallacy, the representative heuristic, or the recogition heuristic, and investigate their relation to rule length and plausibility.