Robust Model Based Reinforcement Learning Using $\mathcal{L}_1$ Adaptive Control
This work addresses robustness issues in MBRL for control tasks, offering an incremental improvement by augmenting existing algorithms with adaptive control techniques.
The paper tackles the problem of enhancing robustness and sample efficiency in Model-Based Reinforcement Learning (MBRL) by introducing $\mathcal{L}_1$-MBRL, a control-theoretic augmentation scheme that perturbs control inputs using $\mathcal{L}_1$ adaptive control to handle uncertainties. The result is improved performance across multiple MuJoCo environments, outperforming original MBRL algorithms with and without system noise.
We introduce $\mathcal{L}_1$-MBRL, a control-theoretic augmentation scheme for Model-Based Reinforcement Learning (MBRL) algorithms. Unlike model-free approaches, MBRL algorithms learn a model of the transition function using data and use it to design a control input. Our approach generates a series of approximate control-affine models of the learned transition function according to the proposed switching law. Using the approximate model, control input produced by the underlying MBRL is perturbed by the $\mathcal{L}_1$ adaptive control, which is designed to enhance the robustness of the system against uncertainties. Importantly, this approach is agnostic to the choice of MBRL algorithm, enabling the use of the scheme with various MBRL algorithms. MBRL algorithms with $\mathcal{L}_1$ augmentation exhibit enhanced performance and sample efficiency across multiple MuJoCo environments, outperforming the original MBRL algorithms, both with and without system noise.