Finding the Near Optimal Policy via Adaptive Reduced Regularization in MDPs
This work addresses a specific bottleneck in reinforcement learning for researchers, offering an incremental improvement in policy optimization efficiency.
The paper tackles the bias in optimal policies of regularized Markov Decision Processes (MDPs) by proposing an adaptive reduction scheme for the regularization coefficient λ, which reduces iteration complexity for obtaining an ε-optimal policy compared to using a fixed small λ.
Regularized MDPs serve as a smooth version of original MDPs. However, biased optimal policy always exists for regularized MDPs. Instead of making the coefficientλof regularized term sufficiently small, we propose an adaptive reduction scheme for λ to approximate optimal policy of the original MDP. It is shown that the iteration complexity for obtaining anε-optimal policy could be reduced in comparison with setting sufficiently smallλ. In addition, there exists strong duality connection between the reduction method and solving the original MDP directly, from which we can derive more adaptive reduction method for certain algorithms.