Behavior-Guided Actor-Critic: Improving Exploration via Learning Policy Behavior Representation for Deep Reinforcement Learning
This addresses exploration inefficiencies in deep RL for agents, offering a method applicable to both stochastic and deterministic actors, though it appears incremental as an enhancement to existing actor-critic frameworks.
The paper tackles the problem of efficient exploration in deep reinforcement learning by proposing Behavior-Guided Actor-Critic (BAC), which uses autoencoders to estimate policy behavior and encourages exploration of less-visited state-action pairs, resulting in considerably better performance compared to several cutting-edge algorithms.
In this work, we propose Behavior-Guided Actor-Critic (BAC), an off-policy actor-critic deep RL algorithm. BAC mathematically formulates the behavior of the policy through autoencoders by providing an accurate estimation of how frequently each state-action pair was visited while taking into consideration state dynamics that play a crucial role in determining the trajectories produced by the policy. The agent is encouraged to change its behavior consistently towards less-visited state-action pairs while attaining good performance by maximizing the expected discounted sum of rewards, resulting in an efficient exploration of the environment and good exploitation of all high reward regions. One prominent aspect of our approach is that it is applicable to both stochastic and deterministic actors in contrast to maximum entropy deep reinforcement learning algorithms. Results show considerably better performances of BAC when compared to several cutting-edge learning algorithms.