Synthesizing Neural Network Controllers with Probabilistic Model based Reinforcement Learning
This work addresses the challenge of data-efficient and scalable controller learning for robotics, particularly in complex domains like underwater vehicles, but it is incremental as it builds upon and refines existing PILCO-based methods.
The authors tackled the problem of learning controllers for robotics systems by improving upon existing model-based reinforcement learning algorithms, achieving data-efficiency competitive with PILCO while enabling optimization of complex neural network controllers. They demonstrated the method's potential by learning motor controllers for a six-legged autonomous underwater vehicle, showing scalability to higher-dimensional and more complex tasks.
We present an algorithm for rapidly learning controllers for robotics systems. The algorithm follows the model-based reinforcement learning paradigm, and improves upon existing algorithms; namely Probabilistic learning in Control (PILCO) and a sample-based version of PILCO with neural network dynamics (Deep-PILCO). We propose training a neural network dynamics model using variational dropout with truncated Log-Normal noise. This allows us to obtain a dynamics model with calibrated uncertainty, which can be used to simulate controller executions via rollouts. We also describe set of techniques, inspired by viewing PILCO as a recurrent neural network model, that are crucial to improve the convergence of the method. We test our method on a variety of benchmark tasks, demonstrating data-efficiency that is competitive with PILCO, while being able to optimize complex neural network controllers. Finally, we assess the performance of the algorithm for learning motor controllers for a six legged autonomous underwater vehicle. This demonstrates the potential of the algorithm for scaling up the dimensionality and dataset sizes, in more complex control tasks.