Discretizing Continuous Action Space with Unimodal Probability Distributions for On-Policy Reinforcement Learning
This work addresses a specific bottleneck in on-policy RL for continuous control, offering an incremental improvement over existing discretization methods.
The paper tackles the problem of high variance in policy gradient estimators when discretizing continuous action spaces for on-policy reinforcement learning by introducing a unimodal discrete policy using Poisson distributions, resulting in significantly faster convergence and higher performance in complex control tasks like Humanoid.
For on-policy reinforcement learning, discretizing action space for continuous control can easily express multiple modes and is straightforward to optimize. However, without considering the inherent ordering between the discrete atomic actions, the explosion in the number of discrete actions can possess undesired properties and induce a higher variance for the policy gradient estimator. In this paper, we introduce a straightforward architecture that addresses this issue by constraining the discrete policy to be unimodal using Poisson probability distributions. This unimodal architecture can better leverage the continuity in the underlying continuous action space using explicit unimodal probability distributions. We conduct extensive experiments to show that the discrete policy with the unimodal probability distribution provides significantly faster convergence and higher performance for on-policy reinforcement learning algorithms in challenging control tasks, especially in highly complex tasks such as Humanoid. We provide theoretical analysis on the variance of the policy gradient estimator, which suggests that our attentively designed unimodal discrete policy can retain a lower variance and yield a stable learning process.