Supported Policy Optimization for Offline Reinforcement Learning
This addresses the challenge of avoiding out-of-distribution actions in offline RL for applications like robotics and autonomous systems, representing a novel method for a known bottleneck.
The paper tackles the problem of offline reinforcement learning by proposing SPOT, a method that uses a VAE-based density estimator to model the behavior policy's support set and integrates it as a pluggable regularization term, achieving state-of-the-art performance on standard benchmarks.
Policy constraint methods to offline reinforcement learning (RL) typically utilize parameterization or regularization that constrains the policy to perform actions within the support set of the behavior policy. The elaborative designs of parameterization methods usually intrude into the policy networks, which may bring extra inference cost and cannot take full advantage of well-established online methods. Regularization methods reduce the divergence between the learned policy and the behavior policy, which may mismatch the inherent density-based definition of support set thereby failing to avoid the out-of-distribution actions effectively. This paper presents Supported Policy OpTimization (SPOT), which is directly derived from the theoretical formalization of the density-based support constraint. SPOT adopts a VAE-based density estimator to explicitly model the support set of behavior policy and presents a simple but effective density-based regularization term, which can be plugged non-intrusively into off-the-shelf off-policy RL algorithms. SPOT achieves the state-of-the-art performance on standard benchmarks for offline RL. Benefiting from the pluggable design, offline pretrained models from SPOT can also be applied to perform online fine-tuning seamlessly.