Towards Practical Credit Assignment for Deep Reinforcement Learning
This work addresses the challenge of making explicit credit assignment methods practical for deep RL, which is important for improving algorithm performance in complex tasks, though it appears incremental as it builds on existing HCA ideas.
The paper tackled the problem of credit assignment in deep reinforcement learning by adapting Hindsight Credit Assignment (HCA) to deep RL environments, proposing modifications that improved performance over Advantage Actor-Critic (A2C) on many Arcade Learning Environment games where accurate credit assignment was crucial.
Credit assignment is a fundamental problem in reinforcement learning, the problem of measuring an action's influence on future rewards. Explicit credit assignment methods have the potential to boost the performance of RL algorithms on many tasks, but thus far remain impractical for general use. Recently, a family of methods called Hindsight Credit Assignment (HCA) was proposed, which explicitly assign credit to actions in hindsight based on the probability of the action having led to an observed outcome. This approach has appealing properties, but remains a largely theoretical idea applicable to a limited set of tabular RL tasks. Moreover, it is unclear how to extend HCA to deep RL environments. In this work, we explore the use of HCA-style credit in a deep RL context. We first describe the limitations of existing HCA algorithms in deep RL that lead to their poor performance or complete lack of training, then propose several theoretically-justified modifications to overcome them. We explore the quantitative and qualitative effects of the resulting algorithm on the Arcade Learning Environment (ALE) benchmark, and observe that it improves performance over Advantage Actor-Critic (A2C) on many games where non-trivial credit assignment is necessary to achieve high scores and where hindsight probabilities can be accurately estimated.