Long-Horizon Q-Learning: Accurate Value Learning via n-Step Inequalities
For researchers and practitioners in reinforcement learning, LQL provides a practical stabilization mechanism that improves long-horizon learning without additional computational overhead.
Long-horizon Q-learning (LQL) introduces a principled backstop against compounding error in off-policy value-based RL by penalizing violations of n-step optimality inequalities, consistently outperforming 1-step and n-step TD methods across online and offline-to-online benchmarks at similar runtime.
Off-policy, value-based reinforcement learning methods such as Q-learning are appealing because they can learn from arbitrary experience, including data collected by older policies or other agents. In practice, however, bootstrapping makes long-horizon learning brittle: estimation errors at later states propagate backward through temporal-difference (TD) updates and can compound over time. We propose long-horizon Q-learning (LQL), which introduces a principled backstop against compounding error when learning the optimal action-value function. LQL builds on a prior optimality tightening observation: any realized action sequence lower-bounds what the optimal policy can achieve in expectation, so acting optimally earlier should not be worse than following the observed actions for several steps before switching to optimal behavior. Our contribution is to turn this inequality into a practical stabilization mechanism for Q-learning by using a hinge loss to penalize violations of these bounds. Importantly, LQL computes these penalties using network outputs already produced for the TD error, requiring no auxiliary networks and no additional forward passes relative to Q-learning. When combined with multiple state-of-the-art methods on a range of online and offline-to-online benchmarks, LQL consistently outperforms both 1-step TD and n-step TD learning at similar runtime.