Learning Finite State Representations of Recurrent Policy Networks
This work addresses the interpretability challenge for RNN policies in control tasks, which is an incremental improvement in making complex models more understandable.
The paper tackles the problem of explaining and analyzing recurrent neural network (RNN) policies in reinforcement and imitation learning by introducing Quantized Bottleneck Insertion to learn finite representations, resulting in surprisingly small representations, such as 3 discrete memory states and 10 observations for a perfect Pong policy, and improved interpretability.
Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.