Optimization Solution Functions as Deterministic Policies for Offline Reinforcement Learning
This work addresses offline RL problems for control applications, offering a novel method to enhance robustness and performance, though it appears incremental in its approach.
The paper tackles the challenges of offline reinforcement learning, such as limited data coverage and value function overestimation, by proposing an implicit actor-critic framework that uses optimization solution functions as deterministic policies, and it shows significant improvement over state-of-the-art methods in real-world applications.
Offline reinforcement learning (RL) is a promising approach for many control applications but faces challenges such as limited data coverage and value function overestimation. In this paper, we propose an implicit actor-critic (iAC) framework that employs optimization solution functions as a deterministic policy (actor) and a monotone function over the optimal value of optimization as a critic. By encoding optimality in the actor policy, we show that the learned policies are robust to the suboptimality of the learned actor parameters via the exponentially decaying sensitivity (EDS) property. We obtain performance guarantees for the proposed iAC framework and show its benefits over general function approximation schemes. Finally, we validate the proposed framework on two real-world applications and show a significant improvement over state-of-the-art (SOTA) offline RL methods.