Time-Constrained Robust MDPs
This work addresses the challenge of deploying reinforcement learning in real-world scenarios with environmental uncertainty, offering a more practical approach for applications like robotics or autonomous systems, though it is incremental in refining existing robust RL frameworks.
The authors tackled the problem of overly conservative policies in robust reinforcement learning due to unrealistic rectangularity assumptions by introducing a time-constrained robust MDP formulation that accounts for correlated and time-dependent disturbances. Their proposed algorithms outperformed traditional deep robust RL methods in time-constrained environments while maintaining robustness in classical benchmarks.
Robust reinforcement learning is essential for deploying reinforcement learning algorithms in real-world scenarios where environmental uncertainty predominates. Traditional robust reinforcement learning often depends on rectangularity assumptions, where adverse probability measures of outcome states are assumed to be independent across different states and actions. This assumption, rarely fulfilled in practice, leads to overly conservative policies. To address this problem, we introduce a new time-constrained robust MDP (TC-RMDP) formulation that considers multifactorial, correlated, and time-dependent disturbances, thus more accurately reflecting real-world dynamics. This formulation goes beyond the conventional rectangularity paradigm, offering new perspectives and expanding the analytical framework for robust RL. We propose three distinct algorithms, each using varying levels of environmental information, and evaluate them extensively on continuous control benchmarks. Our results demonstrate that these algorithms yield an efficient tradeoff between performance and robustness, outperforming traditional deep robust RL methods in time-constrained environments while preserving robustness in classical benchmarks. This study revisits the prevailing assumptions in robust RL and opens new avenues for developing more practical and realistic RL applications.