Efficient Gradient Estimation for Motor Control Learning
This work addresses gradient estimation challenges in motor control learning, offering incremental improvements over existing methods for tasks like robotic arm control.
The paper tackled the problem of gradient estimation errors in reinforcement learning for motor control by presenting two techniques to reduce these errors in the presence of observable input noise, resulting in significant improvements in gradient estimates and learning curves for a simulated dart-throwing task.
The task of estimating the gradient of a function in the presence of noise is central to several forms of reinforcement learning, including policy search methods. We present two techniques for reducing gradient estimation errors in the presence of observable input noise applied to the control signal. The first method extends the idea of a reinforcement baseline by fitting a local linear model to the function whose gradient is being estimated; we show how to find the linear model that minimizes the variance of the gradient estimate, and how to estimate the model from data. The second method improves this further by discounting components of the gradient vector that have high variance. These methods are applied to the problem of motor control learning, where actuator noise has a significant influence on behavior. In particular, we apply the techniques to learn locally optimal controllers for a dart-throwing task using a simulated three-link arm; we demonstrate that proposed methods significantly improve the reward function gradient estimate and, consequently, the learning curve, over existing methods.