Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning
This work addresses computational challenges and suboptimal outcomes in offline RL for applications requiring safe and efficient learning from fixed datasets, though it is incremental as it builds on existing uncertainty-based methods.
The paper tackles the problem of unreliable value estimation and distributional shift in offline reinforcement learning by introducing diverse randomized value functions to estimate Q-value posteriors and apply pessimistic penalties for out-of-distribution actions, resulting in significant performance improvements and enhanced parametric efficiency over baseline methods.
Offline Reinforcement Learning (RL) faces distributional shift and unreliable value estimation, especially for out-of-distribution (OOD) actions. To address this, existing uncertainty-based methods penalize the value function with uncertainty quantification and demand numerous ensemble networks, posing computational challenges and suboptimal outcomes. In this paper, we introduce a novel strategy employing diverse randomized value functions to estimate the posterior distribution of $Q$-values. It provides robust uncertainty quantification and estimates lower confidence bounds (LCB) of $Q$-values. By applying moderate value penalties for OOD actions, our method fosters a provably pessimistic approach. We also emphasize on diversity within randomized value functions and enhance efficiency by introducing a diversity regularization method, reducing the requisite number of networks. These modules lead to reliable value estimation and efficient policy learning from offline data. Theoretical analysis shows that our method recovers the provably efficient LCB-penalty under linear MDP assumptions. Extensive empirical results also demonstrate that our proposed method significantly outperforms baseline methods in terms of performance and parametric efficiency.