Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
This work addresses a key bottleneck in offline RL for researchers and practitioners by improving value estimation, though it is incremental as it builds on existing model-based methods.
The paper tackles the problem of inaccurate value estimation in model-based offline reinforcement learning by introducing Lower Expectile Q-learning (LEQ), which uses lower expectile regression of λ-returns to provide low-bias value estimation, resulting in significant performance improvements on long-horizon tasks like D4RL AntMaze and robust performance across diverse domains including state-based and pixel-based tasks.
Model-based offline reinforcement learning (RL) is a compelling approach that addresses the challenge of learning from limited, static data by generating imaginary trajectories using learned models. However, these approaches often struggle with inaccurate value estimation from model rollouts. In this paper, we introduce a novel model-based offline RL method, Lower Expectile Q-learning (LEQ), which provides a low-bias model-based value estimation via lower expectile regression of $λ$-returns. Our empirical results show that LEQ significantly outperforms previous model-based offline RL methods on long-horizon tasks, such as the D4RL AntMaze tasks, matching or surpassing the performance of model-free approaches and sequence modeling approaches. Furthermore, LEQ matches the performance of state-of-the-art model-based and model-free methods in dense-reward environments across both state-based tasks (NeoRL and D4RL) and pixel-based tasks (V-D4RL), showing that LEQ works robustly across diverse domains. Our ablation studies demonstrate that lower expectile regression, $λ$-returns, and critic training on offline data are all crucial for LEQ.