Interpreting Transformers Through Attention Head Intervention
This addresses the need for interpretability in AI for high-stakes domains and cognitive science, but appears incremental as it builds on existing intervention methods.
The paper tackles the problem of understanding neural mechanisms in transformers by using attention head intervention, aiming to enable accountability, study cognition emergence, and discover new knowledge from AI systems.
Neural networks are growing more capable on their own, but we do not understand their neural mechanisms. Understanding these mechanisms' decision-making processes, or mechanistic interpretability, enables (1) accountability and control in high-stakes domains, (2) the study of digital brains and the emergence of cognition, and (3) discovery of new knowledge when AI systems outperform humans.