Finding patterns in Knowledge Attribution for Transformers
This work provides insights into how knowledge is organized in transformer models, which could aid interpretability for researchers and practitioners, but it is incremental as it builds on existing attribution methods.
The study analyzed the Knowledge Neurons framework to attribute factual and relational knowledge to specific neurons in a 12-layer multi-lingual BERT model, finding that factual knowledge is concentrated in middle and higher layers (≥6), with relational information refined into factual knowledge in the last layers, and that grammatical knowledge is more dispersed than factual knowledge.
We analyze the Knowledge Neurons framework for the attribution of factual and relational knowledge to particular neurons in the transformer network. We use a 12-layer multi-lingual BERT model for our experiments. Our study reveals various interesting phenomena. We observe that mostly factual knowledge can be attributed to middle and higher layers of the network($\ge 6$). Further analysis reveals that the middle layers($6-9$) are mostly responsible for relational information, which is further refined into actual factual knowledge or the "correct answer" in the last few layers($10-12$). Our experiments also show that the model handles prompts in different languages, but representing the same fact, similarly, providing further evidence for effectiveness of multi-lingual pre-training. Applying the attribution scheme for grammatical knowledge, we find that grammatical knowledge is far more dispersed among the neurons than factual knowledge.