Robust Regularized Policy Iteration under Transition Uncertainty
This work addresses the challenge of safe and reliable policy learning in offline RL for applications where online exploration is risky or data is limited, though it is incremental as it builds on existing robust optimization frameworks.
The paper tackles the problem of performance degradation in offline reinforcement learning due to distribution shift and transition uncertainty by formulating it as robust policy optimization and proposing Robust Regularized Policy Iteration (RRPI). The result is that RRPI achieves strong average performance, outperforming recent baselines like PMDB on most D4RL benchmark environments while remaining competitive on others.
Offline reinforcement learning (RL) enables data-efficient and safe policy learning without online exploration, but its performance often degrades under distribution shift. The learned policy may visit out-of-distribution state-action pairs where value estimates and learned dynamics are unreliable. To address policy-induced extrapolation and transition uncertainty in a unified framework, we formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set and optimizing the policy against the worst-case dynamics. We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate and derives an efficient policy iteration procedure based on a robust regularized Bellman operator. We provide theoretical guarantees by showing that the proposed operator is a $γ$-contraction and that iteratively updating the surrogate yields monotonic improvement of the original robust objective with convergence. Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines including percentile-based methods such as PMDB on the majority of environments while remaining competitive on the rest. Moreover, RRPI exhibits robust behavior. The learned $Q$-values decrease in regions with higher epistemic uncertainty, suggesting that the resulting policy avoids unreliable out-of-distribution actions under transition uncertainty.