Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
This work addresses the problem of offline RL for researchers by providing more lax theoretical conditions, though it appears incremental as it builds on existing frameworks with specific refinements.
The paper tackles offline reinforcement learning by proposing value-based algorithms with PAC guarantees under relaxed assumptions of partial coverage and realizability, achieving refined conditions for offline RL.
In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of assuming some coverage, realizability, Bellman completeness, and/or hard margin (gap). In this work we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms to accurately estimate soft or vanilla Q-functions with $L^2$-convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying.