Interpreting Language Models with Contrastive Explanations
This addresses the challenge of providing interpretable explanations for language generation models, which is crucial for developers and users in NLP, though it is incremental as it builds on existing interpretability methods.
The paper tackles the problem of interpreting language models for generation tasks by proposing contrastive explanations that identify salient input tokens to explain why one token was predicted over another, resulting in quantifiably better verification of grammatical phenomena and improved human simulatability.
Model interpretability methods are often used to explain NLP model decisions on tasks such as text classification, where the output space is relatively small. However, when applied to language generation, where the output space often consists of tens of thousands of tokens, these methods are unable to provide informative explanations. Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. To disentangle the different decisions in language modeling, we focus on explaining language models contrastively: we look for salient input tokens that explain why the model predicted one token instead of another. We demonstrate that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena, and that they significantly improve contrastive model simulatability for human observers. We also identify groups of contrastive decisions where the model uses similar evidence, and we are able to characterize what input tokens models use during various language generation decisions.