Uncertainty Aware System Identification with Universal Policies
This addresses a common bottleneck in robotics and control for researchers and practitioners by providing a more efficient method for parameter estimation in sim2real transfer, though it is incremental as it builds on existing techniques like Domain Randomisation.
The paper tackles the problem of estimating real-world environmental parameters for sim2real transfer by proposing Uncertainty-aware policy search (UncAPS), which uses a Universal Policy Network and robust Bayesian optimization to craft robust policies, showing improved performance in noisy, continuous control environments.
Sim2real transfer is primarily concerned with transferring policies trained in simulation to potentially noisy real world environments. A common problem associated with sim2real transfer is estimating the real-world environmental parameters to ground the simulated environment to. Although existing methods such as Domain Randomisation (DR) can produce robust policies by sampling from a distribution of parameters during training, there is no established method for identifying the parameters of the corresponding distribution for a given real-world setting. In this work, we propose Uncertainty-aware policy search (UncAPS), where we use Universal Policy Network (UPN) to store simulation-trained task-specific policies across the full range of environmental parameters and then subsequently employ robust Bayesian optimisation to craft robust policies for the given environment by combining relevant UPN policies in a DR like fashion. Such policy-driven grounding is expected to be more efficient as it estimates only task-relevant sets of parameters. Further, we also account for the estimation uncertainties in the search process to produce policies that are robust against both aleatoric and epistemic uncertainties. We empirically evaluate our approach in a range of noisy, continuous control environments, and show its improved performance compared to competing baselines.