Robust Entropy-regularized Markov Decision Processes
This addresses the problem of policy robustness under uncertainty in reinforcement learning for applications like exploration and imitation learning, but it is incremental as it combines existing schemes.
The paper tackles the sensitivity of entropy-regularized MDP policies to inaccurate transition probability estimates by introducing a robust version that combines robust MDP and entropy-regularized MDP, showing that essential properties hold to make the problem tractable and enabling integration into algorithms like value or policy iteration.
Stochastic and soft optimal policies resulting from entropy-regularized Markov decision processes (ER-MDP) are desirable for exploration and imitation learning applications. Motivated by the fact that such policies are sensitive with respect to the state transition probabilities, and the estimation of these probabilities may be inaccurate, we study a robust version of the ER-MDP model, where the stochastic optimal policies are required to be robust with respect to the ambiguity in the underlying transition probabilities. Our work is at the crossroads of two important schemes in reinforcement learning (RL), namely, robust MDP and entropy regularized MDP. We show that essential properties that hold for the non-robust ER-MDP and robust unregularized MDP models also hold in our settings, making the robust ER-MDP problem tractable. We show how our framework and results can be integrated into different algorithmic schemes including value or (modified) policy iteration, which would lead to new robust RL and inverse RL algorithms to handle uncertainties. Analyses on computational complexity and error propagation under conventional uncertainty settings are also provided.