Interpreting Pretrained Language Models via Concept Bottlenecks
This addresses the problem of interpretability for users of language models, offering a more intuitive approach than previous methods like attention weights, though it is incremental in building on concept-based techniques.
The paper tackles the lack of interpretability in pretrained language models by proposing a method using high-level concepts like 'Food' to analyze model behavior, showing it provides insights, helps diagnose failures, and improves robustness against noisy labels in evaluations on real-world datasets.
Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. However, the lack of interpretability due to their ``black-box'' nature poses challenges for responsible implementation. Although previous studies have attempted to improve interpretability by using, e.g., attention weights in self-attention layers, these weights often lack clarity, readability, and intuitiveness. In this research, we propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans. For example, we learn the concept of ``Food'' and investigate how it influences the prediction of a model's sentiment towards a restaurant review. We introduce C$^3$M, which combines human-annotated and machine-generated concepts to extract hidden neurons designed to encapsulate semantically meaningful and task-specific concepts. Through empirical evaluations on real-world datasets, we manifest that our approach offers valuable insights to interpret PLM behavior, helps diagnose model failures, and enhances model robustness amidst noisy concept labels.