Gradient Q$(σ, λ)$: A Unified Algorithm with Function Approximation for Reinforcement Learning
This work addresses a scalability issue in reinforcement learning for researchers and practitioners, but it is incremental as it extends an existing algorithm with function approximation.
The paper tackles the problem of scaling the Q(σ,λ) algorithm, which unifies full-sampling and pure-expectation methods in reinforcement learning, to large-scale settings by extending it with linear function approximation, resulting in GQ(σ,λ) that achieves better performance than existing methods in standard domains.
Full-sampling (e.g., Q-learning) and pure-expectation (e.g., Expected Sarsa) algorithms are efficient and frequently used techniques in reinforcement learning. Q$(σ,λ)$ is the first approach unifies them with eligibility trace through the sampling degree $σ$. However, it is limited to the tabular case, for large-scale learning, the Q$(σ,λ)$ is too expensive to require a huge volume of tables to accurately storage value functions. To address above problem, we propose a GQ$(σ,λ)$ that extends tabular Q$(σ,λ)$ with linear function approximation. We prove the convergence of GQ$(σ,λ)$. Empirical results on some standard domains show that GQ$(σ,λ)$ with a combination of full-sampling with pure-expectation reach a better performance than full-sampling and pure-expectation methods.