Learning to Mix n-Step Returns: Generalizing lambda-Returns for Deep Reinforcement Learning
This work addresses a bottleneck in deep RL for sequential decision-making tasks by enabling adaptive weighting of returns, though it is incremental as it builds upon existing lambda-returns.
The paper tackles the problem of improving temporal difference learning in deep reinforcement learning by generalizing lambda-returns, proposing Confidence-based Autodidactic Returns (CAR) where the agent learns to weight n-step returns end-to-end, and empirically shows that CAR and lambda-returns outperform n-step returns on A3C in Atari 2600, with specific performance gains reported.
Reinforcement Learning (RL) can model complex behavior policies for goal-directed sequential decision making tasks. A hallmark of RL algorithms is Temporal Difference (TD) learning: value function for the current state is moved towards a bootstrapped target that is estimated using next state's value function. $λ$-returns generalize beyond 1-step returns and strike a balance between Monte Carlo and TD learning methods. While lambda-returns have been extensively studied in RL, they haven't been explored a lot in Deep RL. This paper's first contribution is an exhaustive benchmarking of lambda-returns. Although mathematically tractable, the use of exponentially decaying weighting of n-step returns based targets in lambda-returns is a rather ad-hoc design choice. Our second major contribution is that we propose a generalization of lambda-returns called Confidence-based Autodidactic Returns (CAR), wherein the RL agent learns the weighting of the n-step returns in an end-to-end manner. This allows the agent to learn to decide how much it wants to weigh the n-step returns based targets. In contrast, lambda-returns restrict RL agents to use an exponentially decaying weighting scheme. Autodidactic returns can be used for improving any RL algorithm which uses TD learning. We empirically demonstrate that using sophisticated weighted mixtures of multi-step returns (like CAR and lambda-returns) considerably outperforms the use of n-step returns. We perform our experiments on the Asynchronous Advantage Actor Critic (A3C) algorithm in the Atari 2600 domain.