Proximal Policy Optimization with Continuous Bounded Action Space via the Beta Distribution
This addresses a practical limitation in continuous control tasks for reinforcement learning practitioners, though it is an incremental improvement over existing methods.
The paper tackled the mismatch between Gaussian policies' infinite support and real-world bounded action spaces in reinforcement learning by using a Beta distribution for policy modeling, resulting in improved final expected rewards, faster convergence, and a 63% higher success rate in the CarRacing environment compared to Gaussian policies.
Reinforcement learning methods for continuous control tasks have evolved in recent years generating a family of policy gradient methods that rely primarily on a Gaussian distribution for modeling a stochastic policy. However, the Gaussian distribution has an infinite support, whereas real world applications usually have a bounded action space. This dissonance causes an estimation bias that can be eliminated if the Beta distribution is used for the policy instead, as it presents a finite support. In this work, we investigate how this Beta policy performs when it is trained by the Proximal Policy Optimization (PPO) algorithm on two continuous control tasks from OpenAI gym. For both tasks, the Beta policy is superior to the Gaussian policy in terms of agent's final expected reward, also showing more stability and faster convergence of the training process. For the CarRacing environment with high-dimensional image input, the agent's success rate was improved by 63% over the Gaussian policy.