Dual policy as self-model for planning
This work addresses a specific challenge in reinforcement learning for agents needing efficient planning, but it is incremental as it builds on existing self-model and distillation concepts.
The paper tackles the problem of designing self-models for planning in high-dimensional action spaces by proposing a dual-policy agent that uses a distilled policy network as the self-model, resulting in stabilized training, faster inference, better exploration, and improved behavioral understanding, though it incurs the cost of distilling an additional network.
Planning is a data efficient decision-making strategy where an agent selects candidate actions by exploring possible future states. To simulate future states when there is a high-dimensional action space, the knowledge of one's decision making strategy must be used to limit the number of actions to be explored. We refer to the model used to simulate one's decisions as the agent's self-model. While self-models are implicitly used widely in conjunction with world models to plan actions, it remains unclear how self-models should be designed. Inspired by current reinforcement learning approaches and neuroscience, we explore the benefits and limitations of using a distilled policy network as the self-model. In such dual-policy agents, a model-free policy and a distilled policy are used for model-free actions and planned actions, respectively. Our results on a ecologically relevant, parametric environment indicate that distilled policy network for self-model stabilizes training, has faster inference than using model-free policy, promotes better exploration, and could learn a comprehensive understanding of its own behaviors, at the cost of distilling a new network apart from the model-free policy.