BRPO: Batch Residual Policy Optimization
This addresses the issue of conservative policy learning in batch RL for practitioners, though it is incremental as it builds on existing constraint-based methods.
The paper tackles the problem of batch reinforcement learning being overly conservative by proposing residual policies that allow state-action-dependent deviations from the behavior policy, resulting in BRPO achieving state-of-the-art performance in multiple tasks.
In batch reinforcement learning (RL), one often constrains a learned policy to be close to the behavior (data-generating) policy, e.g., by constraining the learned action distribution to differ from the behavior policy by some maximum degree that is the same at each state. This can cause batch RL to be overly conservative, unable to exploit large policy changes at frequently-visited, high-confidence states without risking poor performance at sparsely-visited states. To remedy this, we propose residual policies, where the allowable deviation of the learned policy is state-action-dependent. We derive a new for RL method, BRPO, which learns both the policy and allowable deviation that jointly maximize a lower bound on policy performance. We show that BRPO achieves the state-of-the-art performance in a number of tasks.