Projection Implicit Q-Learning with Support Constraint for Offline Reinforcement Learning
This addresses a critical challenge in offline RL for researchers and practitioners by improving efficiency and performance, though it is incremental as it builds on existing IQL methods.
The paper tackles the problem of extrapolation errors in offline reinforcement learning by proposing Proj-IQL, which enhances Implicit Q-Learning with a support constraint and multi-step projection, achieving state-of-the-art performance on D4RL benchmarks, particularly in navigation domains.
Offline Reinforcement Learning (RL) faces a critical challenge of extrapolation errors caused by out-of-distribution (OOD) actions. Implicit Q-Learning (IQL) algorithm employs expectile regression to achieve in-sample learning, effectively mitigating the risks associated with OOD actions. However, the fixed hyperparameter in policy evaluation and density-based policy improvement method limit its overall efficiency. In this paper, we propose Proj-IQL, a projective IQL algorithm enhanced with the support constraint. In the policy evaluation phase, Proj-IQL generalizes the one-step approach to a multi-step approach through vector projection, while maintaining in-sample learning and expectile regression framework. In the policy improvement phase, Proj-IQL introduces support constraint that is more aligned with the policy evaluation approach. Furthermore, we theoretically demonstrate that Proj-IQL guarantees monotonic policy improvement and enjoys a progressively more rigorous criterion for superior actions. Empirical results demonstrate the Proj-IQL achieves state-of-the-art performance on D4RL benchmarks, especially in challenging navigation domains.