Time-Scale Separation in Q-Learning: Extending TD($\triangle$) for Action-Value Function Decomposition
This addresses a fundamental challenge in reinforcement learning for tasks requiring long-term planning, though it appears incremental as an extension of existing TD(Δ) methods.
The paper tackles the problem of bias-variance trade-off in Q-Learning for long-term rewards by introducing Q(Δ)-Learning, an extension of TD(Δ) that decomposes action-value functions across multiple time scales, demonstrating improved stability and performance over conventional methods in benchmarks like Atari.
Q-Learning is a fundamental off-policy reinforcement learning (RL) algorithm that has the objective of approximating action-value functions in order to learn optimal policies. Nonetheless, it has difficulties in reconciling bias with variance, particularly in the context of long-term rewards. This paper introduces Q($Δ$)-Learning, an extension of TD($Δ$) for the Q-Learning framework. TD($Δ$) facilitates efficient learning over several time scales by breaking the Q($Δ$)-function into distinct discount factors. This approach offers improved learning stability and scalability, especially for long-term tasks where discounting bias may impede convergence. Our methodology guarantees that each element of the Q($Δ$)-function is acquired individually, facilitating expedited convergence on shorter time scales and enhancing the learning of extended time scales. We demonstrate through theoretical analysis and practical evaluations on standard benchmarks like Atari that Q($Δ$)-Learning surpasses conventional Q-Learning and TD learning methods in both tabular and deep RL environments.