Interpretable Local Tree Surrogate Policies
This addresses the issue of limited trust in policy behavior for users, potentially enabling broader use beyond low-impact tasks, though it appears incremental as it builds on existing surrogate methods.
The paper tackles the problem of interpreting high-dimensional policies like neural networks by proposing predictable policy trees as surrogates, which are human-interpretable and provide quantitative predictions of future behavior, demonstrated on simulated tasks.
High-dimensional policies, such as those represented by neural networks, cannot be reasonably interpreted by humans. This lack of interpretability reduces the trust users have in policy behavior, limiting their use to low-impact tasks such as video games. Unfortunately, many methods rely on neural network representations for effective learning. In this work, we propose a method to build predictable policy trees as surrogates for policies such as neural networks. The policy trees are easily human interpretable and provide quantitative predictions of future behavior. We demonstrate the performance of this approach on several simulated tasks.