Bounded Robustness in Reinforcement Learning via Lexicographic Objectives
This addresses the need for explainable and verifiable robustness in RL policies, which is incremental as it builds on existing policy gradient methods.
The paper tackles the problem of ensuring policy robustness in Reinforcement Learning to observational noise by quantifying and controlling the alterations to optimal policies, proposing a scheme that trades off expected utility for robustness while preserving convergence and sub-optimality.
Policy robustness in Reinforcement Learning may not be desirable at any cost: the alterations caused by robustness requirements from otherwise optimal policies should be explainable, quantifiable and formally verifiable. In this work we study how policies can be maximally robust to arbitrary observational noise by analysing how they are altered by this noise through a stochastic linear operator interpretation of the disturbances, and establish connections between robustness and properties of the noise kernel and of the underlying MDPs. Then, we construct sufficient conditions for policy robustness, and propose a robustness-inducing scheme, applicable to any policy gradient algorithm, that formally trades off expected policy utility for robustness through lexicographic optimisation, while preserving convergence and sub-optimality in the policy synthesis.