Smoothed Action Value Functions for Learning Gaussian Policies
This work addresses the challenge of efficiently training Gaussian policies in continuous control tasks, offering a novel method that improves upon existing approaches, though it appears incremental in nature.
The paper tackles the problem of learning Gaussian policies in reinforcement learning by introducing smoothed action value functions, which satisfy a Bellman equation and enable gradient-based optimization for both mean and covariance parameters. The result is significantly improved performance on standard continuous control benchmarks, though specific numbers are not provided.
State-action value functions (i.e., Q-values) are ubiquitous in reinforcement learning (RL), giving rise to popular algorithms such as SARSA and Q-learning. We propose a new notion of action value defined by a Gaussian smoothed version of the expected Q-value. We show that such smoothed Q-values still satisfy a Bellman equation, making them learnable from experience sampled from an environment. Moreover, the gradients of expected reward with respect to the mean and covariance of a parameterized Gaussian policy can be recovered from the gradient and Hessian of the smoothed Q-value function. Based on these relationships, we develop new algorithms for training a Gaussian policy directly from a learned smoothed Q-value approximator. The approach is additionally amenable to proximal optimization by augmenting the objective with a penalty on KL-divergence from a previous policy. We find that the ability to learn both a mean and covariance during training leads to significantly improved results on standard continuous control benchmarks.