Solving Robust MDPs through No-Regret Dynamics
This addresses the challenge of robustness in reinforcement learning for agents operating in dynamic environments, though it appears incremental as it builds on existing methods like policy gradient and online nonconvex learning.
The paper tackles the problem of solving robust Markov Decision Processes (MDPs) to handle changes in environmental dynamics, and presents a framework using minimax iterative optimization to achieve a robustness improvement of order O(1/T^{1/2}) in the value function.
Reinforcement Learning is a powerful framework for training agents to navigate different situations, but it is susceptible to changes in environmental dynamics. However, solving Markov Decision Processes that are robust to changes is difficult due to nonconvexity and size of action or state spaces. While most works have analyzed this problem by taking different assumptions on the problem, a general and efficient theoretical analysis is still missing. However, we generate a simple framework for improving robustness by solving a minimax iterative optimization problem where a policy player and an environmental dynamics player are playing against each other. Leveraging recent results in online nonconvex learning and techniques from improving policy gradient methods, we yield an algorithm that maximizes the robustness of the Value Function on the order of $\mathcal{O}\left(\frac{1}{T^{\frac{1}{2}}}\right)$ where $T$ is the number of iterations of the algorithm.