Option Encoder: A Framework for Discovering a Policy Basis in Reinforcement Learning
This addresses efficiency and performance issues in hierarchical reinforcement learning for agents, though it appears incremental as it builds on existing option discovery methods.
The paper tackles the problem of redundant options in hierarchical reinforcement learning by proposing Option Encoder, an auto-encoder framework that discovers a policy basis to compress options, reducing computation and improving performance; it demonstrates efficacy on grid-worlds and a Fetch-Reach robotic task.
Option discovery and skill acquisition frameworks are integral to the functioning of a Hierarchically organized Reinforcement learning agent. However, such techniques often yield a large number of options or skills, which can potentially be represented succinctly by filtering out any redundant information. Such a reduction can reduce the required computation while also improving the performance on a target task. In order to compress an array of option policies, we attempt to find a policy basis that accurately captures the set of all options. In this work, we propose Option Encoder, an auto-encoder based framework with intelligently constrained weights, that helps discover a collection of basis policies. The policy basis can be used as a proxy for the original set of skills in a suitable hierarchically organized framework. We demonstrate the efficacy of our method on a collection of grid-worlds and on the high-dimensional Fetch-Reach robotic manipulation task by evaluating the obtained policy basis on a set of downstream tasks.