Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Language
This addresses the issue of misleading explanations in AI for users and developers, but it is incremental as it builds on existing explainability research with a position paper framework.
The paper tackles the problem that language model explainability often diverges from human mental models, leading to harmful rationalization, and argues that explanations must be truthful, complete, and user-adaptive to achieve truthful understanding.
Language models learn and represent language differently than humans; they learn the form and not the meaning. Thus, to assess the success of language model explainability, we need to consider the impact of its divergence from a user's mental model of language. In this position paper, we argue that in order to avoid harmful rationalization and achieve truthful understanding of language models, explanation processes must satisfy three main conditions: (1) explanations have to truthfully represent the model behavior, i.e., have a high fidelity; (2) explanations must be complete, as missing information distorts the truth; and (3) explanations have to take the user's mental model into account, progressively verifying a person's knowledge and adapting their understanding. We introduce a decision tree model to showcase potential reasons why current explanations fail to reach their objectives. We further emphasize the need for human-centered design to explain the model from multiple perspectives, progressively adapting explanations to changing user expectations.