Mechanistic Interpretability of Reinforcement Learning Agents
This work addresses interpretability for RL practitioners, but it is incremental as it builds on existing techniques like saliency mapping.
The paper tackled the problem of understanding how reinforcement learning agents make decisions by analyzing a neural network trained on maze environments, identifying features like walls and pathways and revealing goal misgeneralization biases, such as a tendency to move towards the top right corner without explicit goals.
This paper explores the mechanistic interpretability of reinforcement learning (RL) agents through an analysis of a neural network trained on procedural maze environments. By dissecting the network's inner workings, we identified fundamental features like maze walls and pathways, forming the basis of the model's decision-making process. A significant observation was the goal misgeneralization, where the RL agent developed biases towards certain navigation strategies, such as consistently moving towards the top right corner, even in the absence of explicit goals. Using techniques like saliency mapping and feature mapping, we visualized these biases. We furthered this exploration with the development of novel tools for interactively exploring layer activations.