Total stochastic gradient algorithms and applications in reinforcement learning
This work addresses a methodological gap in reinforcement learning by providing a novel framework for gradient estimation, though it appears incremental as it builds on existing policy gradient theorems.
The paper tackles the underutilization of the total derivative rule in gradient estimation by introducing a visual framework for creating gradient estimators on graphical models, leading to new gradient estimators based on density estimation and a likelihood ratio gradient, which achieve good performance in model-based policy gradient algorithms and help explain the success of PILCO.
Backpropagation and the chain rule of derivatives have been prominent; however, the total derivative rule has not enjoyed the same amount of attention. In this work we show how the total derivative rule leads to an intuitive visual framework for creating gradient estimators on graphical models. In particular, previous "policy gradient theorems" are easily derived. We derive new gradient estimators based on density estimation, as well as a likelihood ratio gradient, which "jumps" to an intermediate node, not directly to the objective function. We evaluate our methods on model-based policy gradient algorithms, achieve good performance, and present evidence towards demystifying the success of the popular PILCO algorithm.