Policy Optimization via Importance Sampling
This addresses an incremental improvement for reinforcement learning practitioners by optimizing trajectory reuse in policy search.
The paper tackles the problem of deciding when to stop optimizing and collect new trajectories in policy optimization by proposing POIS, a model-free algorithm that uses importance sampling with a high-confidence bound, achieving competitive performance on continuous control tasks compared to state-of-the-art methods.
Policy optimization is an effective reinforcement learning approach to solve continuous control tasks. Recent achievements have shown that alternating online and offline optimization is a successful choice for efficient trajectory reuse. However, deciding when to stop optimizing and collect new trajectories is non-trivial, as it requires to account for the variance of the objective function estimate. In this paper, we propose a novel, model-free, policy search algorithm, POIS, applicable in both action-based and parameter-based settings. We first derive a high-confidence bound for importance sampling estimation; then we define a surrogate objective function, which is optimized offline whenever a new batch of trajectories is collected. Finally, the algorithm is tested on a selection of continuous control tasks, with both linear and deep policies, and compared with state-of-the-art policy optimization methods.