Mildly Constrained Evaluation Policy for Offline Reinforcement Learning
This addresses a key bottleneck in offline RL for robotics and simulation tasks, offering a plug-in method to enhance existing algorithms, though it is incremental as it builds on prior constraint-based approaches.
The paper tackles the problem of overly restrictive constraints in offline reinforcement learning by proposing a Mildly Constrained Evaluation Policy (MCEP) that uses different constraints for value estimation and test-time action selection, resulting in significant performance improvements on tasks like D4RL MuJoCo locomotion and robotic manipulation.
Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a \textit{Mildly Constrained Evaluation Policy (MCEP)} for test time inference with a more constrained \textit{target policy} for value estimation. Since the \textit{target policy} has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3BC (Fujimoto & Gu, 2021), AWAC (Nair et al., 2020) and DQL (Wang et al., 2023) algorithms. The empirical results on D4RL MuJoCo locomotion, high-dimensional humanoid and a set of 16 robotic manipulation tasks show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods. The codes are open-sourced at \url{https://github.com/egg-west/MCEP.git}.