Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions
This work addresses the need for deeper insights into LLM inner workings, which is incremental as it builds on prior independent studies of attention and MLPs.
This study tackled the problem of understanding how attention heads and multi-layer perceptron neurons interact in large language models for next-token prediction, revealing that some attention heads recognize specific contexts to activate downstream token-predicting neurons.
Understanding the inner workings of large language models (LLMs) is crucial for advancing their theoretical foundations and real-world applications. While the attention mechanism and multi-layer perceptrons (MLPs) have been studied independently, their interactions remain largely unexplored. This study investigates how attention heads and next-token neurons interact in LLMs to predict new words. We propose a methodology to identify next-token neurons, find prompts that highly activate them, and determine the upstream attention heads responsible. We then generate and evaluate explanations for the activity of these attention heads in an automated manner. Our findings reveal that some attention heads recognize specific contexts relevant to predicting a token and activate a downstream token-predicting neuron accordingly. This mechanism provides a deeper understanding of how attention heads work with MLP neurons to perform next-token prediction. Our approach offers a foundation for further research into the intricate workings of LLMs and their impact on text generation and understanding.