Interpretability in Activation Space Analysis of Transformers: A Focused Survey
It addresses a gap in interpretability research for transformers, focusing on a specific architectural component, but is incremental as it synthesizes existing work rather than introducing new methods.
This survey paper tackles the under-exploration of interpretability in the activation space of feed-forward layers in transformers, reviewing existing methods and identifying future research directions to enhance analysis in this area.
The field of natural language processing has reached breakthroughs with the advent of transformers. They have remained state-of-the-art since then, and there also has been much research in analyzing, interpreting, and evaluating the attention layers and the underlying embedding space. In addition to the self-attention layers, the feed-forward layers in the transformer are a prominent architectural component. From extensive research, we observe that its role is under-explored. We focus on the latent space, known as the Activation Space, that consists of the neuron activations from these feed-forward layers. In this survey paper, we review interpretability methods that examine the learnings that occurred in this activation space. Since there exists only limited research in this direction, we conduct a detailed examination of each work and point out potential future directions of research. We hope our work provides a step towards strengthening activation space analysis.