Reinforcement Learning in an Adaptable Chess Environment for Detecting Human-understandable Concepts
This addresses the interpretability challenge for self-learning agents, which is crucial for ensuring human oversight and robustness in applications like autonomous vehicles, though it appears incremental as it builds on existing probing techniques in a specific domain.
The paper tackles the problem of interpreting black-box deep neural network agents by developing a method to probe the concepts they internalize during training, demonstrated using a chess-playing agent in a resource-efficient environment.
Self-trained autonomous agents developed using machine learning are showing great promise in a variety of control settings, perhaps most remarkably in applications involving autonomous vehicles. The main challenge associated with self-learned agents in the form of deep neural networks, is their black-box nature: it is impossible for humans to interpret deep neural networks. Therefore, humans cannot directly interpret the actions of deep neural network based agents, or foresee their robustness in different scenarios. In this work, we demonstrate a method for probing which concepts self-learning agents internalise in the course of their training. For demonstration, we use a chess playing agent in a fast and light environment developed specifically to be suitable for research groups without access to enormous computational resources or machine learning models.