Contrastive Sparse Autoencoders for Interpreting Planning of Chess-Playing Agents
This work addresses transparency issues in black-box AI systems for sensitive decision-making, though it is incremental in applying existing interpretability methods to a specific domain.
The authors tackled the problem of interpreting multi-step reasoning in chess-playing AI agents by proposing contrastive sparse autoencoders (CSAE), which extract and interpret meaningful concepts from game trajectories, as validated through qualitative analysis and sanity checks.
AI led chess systems to a superhuman level, yet these systems heavily rely on black-box algorithms. This is unsustainable in ensuring transparency to the end-user, particularly when these systems are responsible for sensitive decision-making. Recent interpretability work has shown that the inner representations of Deep Neural Networks (DNNs) were fathomable and contained human-understandable concepts. Yet, these methods are seldom contextualised and are often based on a single hidden state, which makes them unable to interpret multi-step reasoning, e.g. planning. In this respect, we propose contrastive sparse autoencoders (CSAE), a novel framework for studying pairs of game trajectories. Using CSAE, we are able to extract and interpret concepts that are meaningful to the chess-agent plans. We primarily focused on a qualitative analysis of the CSAE features before proposing an automated feature taxonomy. Furthermore, to evaluate the quality of our trained CSAE, we devise sanity checks to wave spurious correlations in our results.