Efficient Duple Perturbation Robustness in Low-rank MDPs
This work addresses robustness challenges in reinforcement learning for applications with large or continuous state-action spaces, representing an incremental improvement over existing methods.
The paper tackles the efficiency issues in robust reinforcement learning by introducing duple perturbation robustness for low-rank MDPs, resulting in a provably efficient algorithm with theoretical convergence guarantees.
The pursuit of robustness has recently been a popular topic in reinforcement learning (RL) research, yet the existing methods generally suffer from efficiency issues that obstruct their real-world implementation. In this paper, we introduce duple perturbation robustness, i.e. perturbation on both the feature and factor vectors for low-rank Markov decision processes (MDPs), via a novel characterization of $(ξ,η)$-ambiguity sets. The novel robust MDP formulation is compatible with the function representation view, and therefore, is naturally applicable to practical RL problems with large or even continuous state-action spaces. Meanwhile, it also gives rise to a provably efficient and practical algorithm with theoretical convergence rate guarantee. Examples are designed to justify the new robustness concept, and algorithmic efficiency is supported by both theoretical bounds and numerical simulations.