Efficient Baseline-free Sampling in Parameter Exploring Policy Gradients: Super Symmetric PGPE
This is an incremental improvement for reinforcement learning practitioners using parameter-exploring policy gradients.
The paper tackles the problem of asymmetric reward distributions in Policy Gradients with Parameter-based Exploration (PGPE) by proposing a method to sample exploration parameters quasi-symmetrically, demonstrating that this approach is superior in sample efficiency and robustness compared to the original method.
Policy Gradient methods that explore directly in parameter space are among the most effective and robust direct policy search methods and have drawn a lot of attention lately. The basic method from this field, Policy Gradients with Parameter-based Exploration, uses two samples that are symmetric around the current hypothesis to circumvent misleading reward in \emph{asymmetrical} reward distributed problems gathered with the usual baseline approach. The exploration parameters are still updated by a baseline approach - leaving the exploration prone to asymmetric reward distributions. In this paper we will show how the exploration parameters can be sampled quasi symmetric despite having limited instead of free parameters for exploration. We give a transformation approximation to get quasi symmetric samples with respect to the exploration without changing the overall sampling distribution. Finally we will demonstrate that sampling symmetrically also for the exploration parameters is superior in needs of samples and robustness than the original sampling approach.