GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values
This addresses a fundamental contradiction in offline RL estimation for researchers, but it is incremental as it builds directly on prior work.
The paper tackles the problem of estimating density ratios in off-policy reinforcement learning by introducing GradientDICE, which fixes convergence issues in the state-of-the-art GenDICE method, resulting in provable convergence under linear function approximation.
We present GradientDICE for estimating the density ratio between the state distribution of the target policy and the sampling distribution in off-policy reinforcement learning. GradientDICE fixes several problems of GenDICE (Zhang et al., 2020), the state-of-the-art for estimating such density ratios. Namely, the optimization problem in GenDICE is not a convex-concave saddle-point problem once nonlinearity in optimization variable parameterization is introduced to ensure positivity, so any primal-dual algorithm is not guaranteed to converge or find the desired solution. However, such nonlinearity is essential to ensure the consistency of GenDICE even with a tabular representation. This is a fundamental contradiction, resulting from GenDICE's original formulation of the optimization problem. In GradientDICE, we optimize a different objective from GenDICE by using the Perron-Frobenius theorem and eliminating GenDICE's use of divergence. Consequently, nonlinearity in parameterization is not necessary for GradientDICE, which is provably convergent under linear function approximation.