Trust Region Policy Optimization
This provides a practical method for stable policy optimization in reinforcement learning, addressing issues like hyperparameter sensitivity for researchers and practitioners.
The authors tackled the problem of optimizing policies in reinforcement learning with guaranteed monotonic improvement, resulting in the Trust Region Policy Optimization (TRPO) algorithm that demonstrated robust performance on tasks like robotic gaits and Atari games with minimal hyperparameter tuning.
We describe an iterative procedure for optimizing policies, with guaranteed monotonic improvement. By making several approximations to the theoretically-justified procedure, we develop a practical algorithm, called Trust Region Policy Optimization (TRPO). This algorithm is similar to natural policy gradient methods and is effective for optimizing large nonlinear policies such as neural networks. Our experiments demonstrate its robust performance on a wide variety of tasks: learning simulated robotic swimming, hopping, and walking gaits; and playing Atari games using images of the screen as input. Despite its approximations that deviate from the theory, TRPO tends to give monotonic improvement, with little tuning of hyperparameters.