On- and Off-Policy Monotonic Policy Improvement
This work addresses a fundamental problem in reinforcement learning for improving policy stability and efficiency, though it appears incremental as it builds on existing trust region and natural policy gradient methods.
The paper tackles the challenge of ensuring monotonic policy improvement in reinforcement learning from both on- and off-policy samples by deriving a lower bound on performance differences, resulting in a method that shows performance in two benchmark problems.
Monotonic policy improvement and off-policy learning are two main desirable properties for reinforcement learning algorithms. In this paper, by lower bounding the performance difference of two policies, we show that the monotonic policy improvement is guaranteed from on- and off-policy mixture samples. An optimization procedure which applies the proposed bound can be regarded as an off-policy natural policy gradient method. In order to support the theoretical result, we provide a trust region policy optimization method using experience replay as a naive application of our bound, and evaluate its performance in two classical benchmark problems.