Acceleration in Policy Optimization
This work addresses the challenge of improving convergence speed in reinforcement learning for researchers and practitioners, though it appears incremental as it builds upon existing methods without claiming broad SOTA gains.
The paper tackles the problem of accelerating policy optimization in reinforcement learning by integrating foresight through optimistic and adaptive updates, resulting in a unifying paradigm that connects various existing algorithms and demonstrates design choices for acceleration in an illustrative task.
We work towards a unifying paradigm for accelerating policy optimization methods in reinforcement learning (RL) by integrating foresight in the policy improvement step via optimistic and adaptive updates. Leveraging the connection between policy iteration and policy gradient methods, we view policy optimization algorithms as iteratively solving a sequence of surrogate objectives, local lower bounds on the original objective. We define optimism as predictive modelling of the future behavior of a policy, and adaptivity as taking immediate and anticipatory corrective actions to mitigate accumulating errors from overshooting predictions or delayed responses to change. We use this shared lens to jointly express other well-known algorithms, including model-based policy improvement based on forward search, and optimistic meta-learning algorithms. We analyze properties of this formulation, and show connections to other accelerated optimization algorithms. Then, we design an optimistic policy gradient algorithm, adaptive via meta-gradient learning, and empirically highlight several design choices pertaining to acceleration, in an illustrative task.