CGAR: Critic Guided Action Redistribution in Reinforcement Leaning
This addresses sample efficiency for game-playing RL agents, though it appears incremental as it builds on existing off-policy actor-critic methods.
The paper tackles inefficient random exploration in reinforcement learning by proposing the Critic Guided Action Redistribution (CGAR) algorithm, which uses critic Q-values to redistribute actions sampled from the actor policy, and demonstrates improved sample efficiency and state-of-the-art performance on OpenAI MuJoCo tasks.
Training a game-playing reinforcement learning agent requires multiple interactions with the environment. Ignorant random exploration may cause a waste of time and resources. It's essential to alleviate such waste. As discussed in this paper, under the settings of the off-policy actor critic algorithms, we demonstrate that the critic can bring more expected discounted rewards than or at least equal to the actor. Thus, the Q value predicted by the critic is a better signal to redistribute the action originally sampled from the policy distribution predicted by the actor. This paper introduces the novel Critic Guided Action Redistribution (CGAR) algorithm and tests it on the OpenAI MuJoCo tasks. The experimental results demonstrate that our method improves the sample efficiency and achieves state-of-the-art performance. Our code can be found at https://github.com/tairanhuang/CGAR.