Automatically Finding Rule-Based Neurons in OthelloGPT
This work addresses interpretability in AI for researchers, providing a tool to map game behaviors to neurons, but it is incremental as it builds on existing methods in a specific domain.
The researchers tackled the problem of interpreting neurons in OthelloGPT by developing an automated method using decision trees to identify rule-based neurons, finding that about half of the neurons in layer 5 could be described with high accuracy (R^2 > 0.7 for 913 of 2,048 neurons) and verifying causal relevance through interventions that showed a 5-10 fold degradation in model performance for targeted patterns.
OthelloGPT, a transformer trained to predict valid moves in Othello, provides an ideal testbed for interpretability research. The model is complex enough to exhibit rich computational patterns, yet grounded in rule-based game logic that enables meaningful reverse-engineering. We present an automated approach based on decision trees to identify and interpret MLP neurons that encode rule-based game logic. Our method trains regression decision trees to map board states to neuron activations, then extracts decision paths where neurons are highly active to convert them into human-readable logical forms. These descriptions reveal highly interpretable patterns; for instance, neurons that specifically detect when diagonal moves become legal. Our findings suggest that roughly half of the neurons in layer 5 can be accurately described by compact, rule-based decision trees ($R^2 > 0.7$ for 913 of 2,048 neurons), while the remainder likely participate in more distributed or non-rule-based computations. We verify the causal relevance of patterns identified by our decision trees through targeted interventions. For a specific square, for specific game patterns, we ablate neurons corresponding to those patterns and find an approximately 5-10 fold stronger degradation in the model's ability to predict legal moves along those patterns compared to control patterns. To facilitate future work, we provide a Python tool that maps rule-based game behaviors to their implementing neurons, serving as a resource for researchers to test whether their interpretability methods recover meaningful computational structures.