Towards Generating Informative Textual Description for Neurons in Language Models
This addresses the challenge of neuron explainability in language models for researchers and practitioners, offering a scalable and unsupervised method to interpret model internals, though it is incremental as it builds on existing explainability approaches.
The paper tackles the problem of understanding what information is captured by neurons in transformer-based language models like BERT, proposing a novel framework that uses generative language models to automatically generate human-interpretable textual descriptions for neurons without manual annotations, achieving 75% precision@2 and 50% recall@2 in experiments.
Recent developments in transformer-based language models have allowed them to capture a wide variety of world knowledge that can be adapted to downstream tasks with limited resources. However, what pieces of information are understood in these models is unclear, and neuron-level contributions in identifying them are largely unknown. Conventional approaches in neuron explainability either depend on a finite set of pre-defined descriptors or require manual annotations for training a secondary model that can then explain the neurons of the primary model. In this paper, we take BERT as an example and we try to remove these constraints and propose a novel and scalable framework that ties textual descriptions to neurons. We leverage the potential of generative language models to discover human-interpretable descriptors present in a dataset and use an unsupervised approach to explain neurons with these descriptors. Through various qualitative and quantitative analyses, we demonstrate the effectiveness of this framework in generating useful data-specific descriptors with little human involvement in identifying the neurons that encode these descriptors. In particular, our experiment shows that the proposed approach achieves 75% precision@2, and 50% recall@2