Challenges in Mechanistically Interpreting Model Representations
This work addresses the problem of improving safety and trust in AI for researchers and practitioners by highlighting gaps in mechanistic interpretability, though it is incremental as it builds on existing MI frameworks.
The paper tackles the challenge of mechanistically interpreting complex AI model representations, particularly for non-trivial capabilities like dishonesty, by formalizing representation analysis and demonstrating current methods' insufficiency through an exploratory study on Mistral-7B-Instruct-v0.1.
Mechanistic interpretability (MI) aims to understand AI models by reverse-engineering the exact algorithms neural networks learn. Most works in MI so far have studied behaviors and capabilities that are trivial and token-aligned. However, most capabilities important for safety and trust are not that trivial, which advocates for the study of hidden representations inside these networks as the unit of analysis. We formalize representations for features and behaviors, highlight their importance and evaluation, and perform an exploratory study of dishonesty representations in `Mistral-7B-Instruct-v0.1'. We justify that studying representations is an important and under-studied field, and highlight several challenges that arise while attempting to do so through currently established methods in MI, showing their insufficiency and advocating work on new frameworks for the same.