Virtual vs. Real: Trading Off Simulations and Physical Experiments in Reinforcement Learning with Bayesian Optimization
This work addresses the time-consuming and costly process of manual policy tuning in robotics, offering a cost-effective solution for practitioners, though it is incremental as it extends an existing algorithm to multiple information sources.
The paper tackles the problem of reducing the number of physical experiments needed to tune control policies in reinforcement learning by combining cheap but inaccurate simulations with expensive and accurate physical experiments using a Bayesian optimization method. The result is a method that finds good control policies with fewer experiments than standard approaches, as confirmed in a cart-pole system.
In practice, the parameters of control policies are often tuned manually. This is time-consuming and frustrating. Reinforcement learning is a promising alternative that aims to automate this process, yet often requires too many experiments to be practical. In this paper, we propose a solution to this problem by exploiting prior knowledge from simulations, which are readily available for most robotic platforms. Specifically, we extend Entropy Search, a Bayesian optimization algorithm that maximizes information gain from each experiment, to the case of multiple information sources. The result is a principled way to automatically combine cheap, but inaccurate information from simulations with expensive and accurate physical experiments in a cost-effective manner. We apply the resulting method to a cart-pole system, which confirms that the algorithm can find good control policies with fewer experiments than standard Bayesian optimization on the physical system only.