Sparse Interventions in Language Models with Differentiable Masking
This work addresses the need for more precise and interpretable neuron-level analysis in language models, though it is incremental as it builds on causal mediation analysis and focuses on specific linguistic tasks.
The paper tackles the problem of identifying small subsets of neurons in language models responsible for specific linguistic phenomena, such as subject-verb agreement and gender bias, by proposing a method using differentiable masking with L0 regularization, which finds sparse solutions faster and better than alternatives like REINFORCE.
There has been a lot of interest in understanding what information is captured by hidden representations of language models (LMs). Typically, interpretation methods i) do not guarantee that the model actually uses the encoded information, and ii) do not discover small subsets of neurons responsible for a considered phenomenon. Inspired by causal mediation analysis, we propose a method that discovers within a neural LM a small subset of neurons responsible for a particular linguistic phenomenon, i.e., subsets causing a change in the corresponding token emission probabilities. We use a differentiable relaxation to approximately search through the combinatorial space. An $L_0$ regularization term ensures that the search converges to discrete and sparse solutions. We apply our method to analyze subject-verb number agreement and gender bias detection in LSTMs. We observe that it is fast and finds better solutions than the alternative (REINFORCE). Our experiments confirm that each of these phenomenons is mediated through a small subset of neurons that do not play any other discernible role.