Quantile Q-Learning: Revisiting Offline Extreme Q-Learning with Quantile Regression
This addresses the problem of unreliable offline RL training for high-risk applications, though it is incremental as it builds on existing Extreme Q-Learning methods.
The paper tackled the instability and hyperparameter sensitivity of Extreme Q-Learning in offline reinforcement learning by proposing a method to estimate the temperature coefficient via quantile regression and adding value regularization, achieving competitive or superior performance on benchmarks like D4RL and NeoRL2 with stable training and consistent hyperparameters.
Offline reinforcement learning (RL) enables policy learning from fixed datasets without further environment interaction, making it particularly valuable in high-risk or costly domains. Extreme $Q$-Learning (XQL) is a recent offline RL method that models Bellman errors using the Extreme Value Theorem, yielding strong empirical performance. However, XQL and its stabilized variant MXQL suffer from notable limitations: both require extensive hyperparameter tuning specific to each dataset and domain, and also exhibit instability during training. To address these issues, we proposed a principled method to estimate the temperature coefficient $β$ via quantile regression under mild assumptions. To further improve training stability, we introduce a value regularization technique with mild generalization, inspired by recent advances in constrained value learning. Experimental results demonstrate that the proposed algorithm achieves competitive or superior performance across a range of benchmark tasks, including D4RL and NeoRL2, while maintaining stable training dynamics and using a consistent set of hyperparameters across all datasets and domains.