Distributionally Robust Policy Learning via Adversarial Environment Generation
This addresses the challenge of robust policy learning for robotics and control systems, offering a novel approach to handle realistic distribution shifts, though it builds incrementally on existing DRO and generative methods.
The paper tackles the problem of training control policies that generalize to unseen environments by proposing DRAGEN, a method that uses adversarial environment generation within a distributionally robust optimization framework, resulting in strong out-of-distribution generalization in simulations and improved sim2real performance in hardware grasping experiments compared to domain randomization.
Our goal is to train control policies that generalize well to unseen environments. Inspired by the Distributionally Robust Optimization (DRO) framework, we propose DRAGEN - Distributionally Robust policy learning via Adversarial Generation of ENvironments - for iteratively improving robustness of policies to realistic distribution shifts by generating adversarial environments. The key idea is to learn a generative model for environments whose latent variables capture cost-predictive and realistic variations in environments. We perform DRO with respect to a Wasserstein ball around the empirical distribution of environments by generating realistic adversarial environments via gradient ascent on the latent space. We demonstrate strong Out-of-Distribution (OoD) generalization in simulation for (i) swinging up a pendulum with onboard vision and (ii) grasping realistic 3D objects. Grasping experiments on hardware demonstrate better sim2real performance compared to domain randomization.