Taylor Expansion Policy Optimization
This work addresses policy optimization challenges in reinforcement learning, offering a novel formalism that enhances existing algorithms, though it appears incremental as it builds on prior methods.
The paper tackles policy optimization in reinforcement learning by proposing a Taylor expansion-based formalism that generalizes prior methods like TRPO as a first-order case and relates to off-policy evaluation, resulting in performance improvements for several state-of-the-art distributed algorithms.
In this work, we investigate the application of Taylor expansions in reinforcement learning. In particular, we propose Taylor expansion policy optimization, a policy optimization formalism that generalizes prior work (e.g., TRPO) as a first-order special case. We also show that Taylor expansions intimately relate to off-policy evaluation. Finally, we show that this new formulation entails modifications which improve the performance of several state-of-the-art distributed algorithms.