Bayesian Soft Actor-Critic: A Directed Acyclic Strategy Graph Based Deep Reinforcement Learning
This work addresses policy efficiency for reinforcement learning agents in hazardous and dynamic settings, representing an incremental improvement over existing methods like SAC.
The paper tackles the challenge of designing efficient policies for agents in complex environments by proposing Bayesian Soft Actor-Critic (BSAC), which decomposes policies into simpler sub-policies using a directed acyclic strategy graph and Bayesian chaining, resulting in significantly improved training efficiency on standard continuous control benchmarks.
Adopting reasonable strategies is challenging but crucial for an intelligent agent with limited resources working in hazardous, unstructured, and dynamic environments to improve the system's utility, decrease the overall cost, and increase mission success probability. This paper proposes a novel directed acyclic strategy graph decomposition approach based on Bayesian chaining to separate an intricate policy into several simple sub-policies and organize their relationships as Bayesian strategy networks (BSN). We integrate this approach into the state-of-the-art DRL method -- soft actor-critic (SAC), and build the corresponding Bayesian soft actor-critic (BSAC) model by organizing several sub-policies as a joint policy. We compare our method against the state-of-the-art deep reinforcement learning algorithms on the standard continuous control benchmarks in the OpenAI Gym environment. The results demonstrate that the promising potential of the BSAC method significantly improves training efficiency.