Potential-Based Advice for Stochastic Policy Learning
This work addresses the challenge of efficient policy learning in reinforcement learning, offering incremental improvements for domains like control tasks.
The paper tackles the problem of accelerating reinforcement learning by augmenting rewards with potential functions to preserve optimality in stochastic policies, showing that agents learn faster and achieve higher average rewards in grid world and mountain car environments.
This paper augments the reward received by a reinforcement learning agent with potential functions in order to help the agent learn (possibly stochastic) optimal policies. We show that a potential-based reward shaping scheme is able to preserve optimality of stochastic policies, and demonstrate that the ability of an agent to learn an optimal policy is not affected when this scheme is augmented to soft Q-learning. We propose a method to impart potential based advice schemes to policy gradient algorithms. An algorithm that considers an advantage actor-critic architecture augmented with this scheme is proposed, and we give guarantees on its convergence. Finally, we evaluate our approach on a puddle-jump grid world with indistinguishable states, and the continuous state and action mountain car environment from classical control. Our results indicate that these schemes allow the agent to learn a stochastic optimal policy faster and obtain a higher average reward.