Feature Interactions Reveal Linguistic Structure in Language Models
This work addresses the problem of interpretability in neural networks for researchers, but it is incremental as it builds on existing methods without introducing a new paradigm.
The paper tackled the challenge of evaluating feature interaction attribution methods for interpretability by training models on a formal language classification task, showing that some methods can uncover grammatical rules acquired by models, and extended this to provide insights into linguistic structure in language models.
We study feature interactions in the context of feature attribution methods for post-hoc interpretability. In interpretability research, getting to grips with feature interactions is increasingly recognised as an important challenge, because interacting features are key to the success of neural networks. Feature interactions allow a model to build up hierarchical representations for its input, and might provide an ideal starting point for the investigation into linguistic structure in language models. However, uncovering the exact role that these interactions play is also difficult, and a diverse range of interaction attribution methods has been proposed. In this paper, we focus on the question which of these methods most faithfully reflects the inner workings of the target models. We work out a grey box methodology, in which we train models to perfection on a formal language classification task, using PCFGs. We show that under specific configurations, some methods are indeed able to uncover the grammatical rules acquired by a model. Based on these findings we extend our evaluation to a case study on language models, providing novel insights into the linguistic structure that these models have acquired.