A Strategy-Oriented Bayesian Soft Actor-Critic Model
This work addresses the problem of efficient strategy learning for agents in hazardous and dynamic environments, representing an incremental improvement over existing deep reinforcement learning methods.
The paper tackles the challenge of enabling intelligent agents to adopt reasonable strategies in complex environments by proposing a hierarchical strategy decomposition approach integrated into the soft actor-critic (SAC) method, resulting in the Bayesian soft actor-critic (BSAC) model that significantly improves training efficiency on standard continuous control benchmarks.
Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system's utility, decrease the overall cost, and increase mission success probability. This paper proposes a novel hierarchical strategy decomposition approach based on the Bayesian chain rule to separate an intricate policy into several simple sub-policies and organize their relationships as Bayesian strategy networks (BSN). We integrate this approach into the state-of-the-art DRL method -- soft actor-critic (SAC) and build the corresponding Bayesian soft actor-critic (BSAC) model by organizing several sub-policies as a joint policy. We compare the proposed BSAC method with the SAC and other state-of-the-art approaches such as TD3, DDPG, and PPO on the standard continuous control benchmarks -- Hopper-v2, Walker2d-v2, and Humanoid-v2 -- in MuJoCo with the OpenAI Gym environment. The results demonstrate that the promising potential of the BSAC method significantly improves training efficiency.