Is Q-learning an Ill-posed Problem?
This addresses a critical problem for reinforcement learning practitioners by revealing fundamental limitations in Q-learning, making it an incremental but important critique.
The paper investigates Q-learning instability in continuous environments, finding that even in simple benchmarks, the task of learning a Q-function from policy-specific targets can be inherently ill-posed and prone to failure, casting doubt on its reliability as a universal RL solution.
This paper investigates the instability of Q-learning in continuous environments, a challenge frequently encountered by practitioners. Traditionally, this instability is attributed to bootstrapping and regression model errors. Using a representative reinforcement learning benchmark, we systematically examine the effects of bootstrapping and model inaccuracies by incrementally eliminating these potential error sources. Our findings reveal that even in relatively simple benchmarks, the fundamental task of Q-learning - iteratively learning a Q-function from policy-specific target values - can be inherently ill-posed and prone to failure. These insights cast doubt on the reliability of Q-learning as a universal solution for reinforcement learning problems.