Policy Gradient with Kernel Quadrature
This work addresses efficiency issues in policy gradient methods for reinforcement learning practitioners, though it appears incremental as it builds on existing kernel methods.
The paper tackles the bottleneck of reward evaluation in reinforcement learning by selecting a small, representative subset of episodes for reward computation, using a Gaussian process and kernel quadrature method, and demonstrates this approach in MuJoCo tasks with improved efficiency.
Reward evaluation of episodes becomes a bottleneck in a broad range of reinforcement learning tasks. Our aim in this paper is to select a small but representative subset of a large batch of episodes, only on which we actually compute rewards for more efficient policy gradient iterations. We build a Gaussian process modeling of discounted returns or rewards to derive a positive definite kernel on the space of episodes, run an ``episodic" kernel quadrature method to compress the information of sample episodes, and pass the reduced episodes to the policy network for gradient updates. We present the theoretical background of this procedure as well as its numerical illustrations in MuJoCo tasks.