Distorted Distributional Policy Evaluation for Offline Reinforcement Learning
This work addresses a key limitation in offline reinforcement learning for applications requiring robust generalization, though it is incremental in nature.
The paper tackles the problem of overly conservative value estimates in offline distributional reinforcement learning by introducing quantile distortion, which adjusts pessimism based on data availability, leading to improved performance over uniform pessimism methods.
While Distributional Reinforcement Learning (DRL) methods have demonstrated strong performance in online settings, its success in offline scenarios remains limited. We hypothesize that a key limitation of existing offline DRL methods lies in their approach to uniformly underestimate return quantiles. This uniform pessimism can lead to overly conservative value estimates, ultimately hindering generalization and performance. To address this, we introduce a novel concept called quantile distortion, which enables non-uniform pessimism by adjusting the degree of conservatism based on the availability of supporting data. Our approach is grounded in theoretical analysis and empirically validated, demonstrating improved performance over uniform pessimism.