A Behavior Regularized Implicit Policy for Offline Reinforcement Learning
This work addresses the challenge of training effective agents from static datasets in offline reinforcement learning, which is incremental as it builds on existing policy-matching methods with novel regularization techniques.
The paper tackles the problem of offline reinforcement learning by proposing a framework for learning a flexible yet well-regularized implicit policy, which addresses issues like vulnerability to out-of-distribution state-action pairs and missing rewarding actions. The result is validated through extensive experiments on the D4RL benchmark, showing effectiveness in improving agent performance.
Offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. The lack of environmental interactions makes the policy training vulnerable to state-action pairs far from the training dataset and prone to missing rewarding actions. For training more effective agents, we propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. We further propose a simple modification to the classical policy-matching methods for regularizing with respect to the dual form of the Jensen--Shannon divergence and the integral probability metrics. We theoretically show the correctness of the policy-matching approach, and the correctness and a good finite-sample property of our modification. An effective instantiation of our framework through the GAN structure is provided, together with techniques to explicitly smooth the state-action mapping for robust generalization beyond the static dataset. Extensive experiments and ablation study on the D4RL benchmark validate our framework and the effectiveness of our algorithmic designs.