Hybrid Actor-Critic Reinforcement Learning in Parameterized Action Space
This addresses a specific problem in reinforcement learning for domains with structured action spaces, but it is incremental as it builds on existing actor-critic methods like PPO.
The paper tackles reinforcement learning in parameterized action spaces by proposing a hybrid actor-critic architecture with parallel sub-actor networks, and it demonstrates superior performance over previous methods in experiments on such tasks.
In this paper we propose a hybrid architecture of actor-critic algorithms for reinforcement learning in parameterized action space, which consists of multiple parallel sub-actor networks to decompose the structured action space into simpler action spaces along with a critic network to guide the training of all sub-actor networks. While this paper is mainly focused on parameterized action space, the proposed architecture, which we call hybrid actor-critic, can be extended for more general action spaces which has a hierarchical structure. We present an instance of the hybrid actor-critic architecture based on proximal policy optimization (PPO), which we refer to as hybrid proximal policy optimization (H-PPO). Our experiments test H-PPO on a collection of tasks with parameterized action space, where H-PPO demonstrates superior performance over previous methods of parameterized action reinforcement learning.