Deep Reinforcement Learning Discovers Internal Models
This provides a tool for researchers to analyze DRL policies, but it is incremental as it builds on existing modeling approaches.
The paper tackles the problem of analyzing deep reinforcement learning (DRL) policies by introducing the Semi-Aggregated MDP (SAMDP) model, which identifies skills in agents like DQN and leads to an extra performance gain.
Deep Reinforcement Learning (DRL) is a trending field of research, showing great promise in challenging problems such as playing Atari, solving Go and controlling robots. While DRL agents perform well in practice we are still lacking the tools to analayze their performance. In this work we present the Semi-Aggregated MDP (SAMDP) model. A model best suited to describe policies exhibiting both spatial and temporal hierarchies. We describe its advantages for analyzing trained policies over other modeling approaches, and show that under the right state representation, like that of DQN agents, SAMDP can help to identify skills. We detail the automatic process of creating it from recorded trajectories, up to presenting it on t-SNE maps. We explain how to evaluate its fitness and show surprising results indicating high compatibility with the policy at hand. We conclude by showing how using the SAMDP model, an extra performance gain can be squeezed from the agent.