Average-DICE: Stationary Distribution Correction by Regression
This addresses stability and accuracy issues in reinforcement learning for researchers and practitioners, though it is incremental as it builds on existing density-ratio correction methods.
The paper tackles the problem of stationary state distribution mismatch in off-policy policy evaluation by introducing AVG-DICE, a computationally simple Monte Carlo estimator for density ratios, which is at least as accurate as state-of-the-art methods and sometimes offers orders-of-magnitude improvements in accuracy.
Off-policy policy evaluation (OPE), an essential component of reinforcement learning, has long suffered from stationary state distribution mismatch, undermining both stability and accuracy of OPE estimates. While existing methods correct distribution shifts by estimating density ratios, they often rely on expensive optimization or backward Bellman-based updates and struggle to outperform simpler baselines. We introduce AVG-DICE, a computationally simple Monte Carlo estimator for the density ratio that averages discounted importance sampling ratios, providing an unbiased and consistent correction. AVG-DICE extends naturally to nonlinear function approximation using regression, which we roughly tune and test on OPE tasks based on Mujoco Gym environments and compare with state-of-the-art density-ratio estimators using their reported hyperparameters. In our experiments, AVG-DICE is at least as accurate as state-of-the-art estimators and sometimes offers orders-of-magnitude improvements. However, a sensitivity analysis shows that best-performing hyperparameters may vary substantially across different discount factors, so a re-tuning is suggested.