Understanding and Controlling a Maze-Solving Policy Network
This work provides insights into goal-direction in trained policy networks, which is incremental for understanding AI systems in reinforcement learning.
The researchers studied a pretrained reinforcement learning policy for maze-solving to understand its goal representations, finding that the network pursues multiple context-dependent goals and contains circuits tracking goal locations. They demonstrated partial control over the policy by modifying specific channels, revealing redundant and distributed goal representations.
To understand the goals and goal representations of AI systems, we carefully study a pretrained reinforcement learning policy that solves mazes by navigating to a range of target squares. We find this network pursues multiple context-dependent goals, and we further identify circuits within the network that correspond to one of these goals. In particular, we identified eleven channels that track the location of the goal. By modifying these channels, either with hand-designed interventions or by combining forward passes, we can partially control the policy. We show that this network contains redundant, distributed, and retargetable goal representations, shedding light on the nature of goal-direction in trained policy networks.