SOPE: Spectrum of Off-Policy Estimators
This work addresses the challenge of reliable policy evaluation in high-stakes domains, offering a flexible method to balance bias and variance, though it is incremental in nature.
The paper tackles the bias-variance trade-off in off-policy evaluation for sequential decision making by introducing a spectrum of estimators between trajectory-based importance sampling and state-action visitation distribution methods, showing that these estimators can achieve lower mean-squared error than existing approaches.
Many sequential decision making problems are high-stakes and require off-policy evaluation (OPE) of a new policy using historical data collected using some other policy. One of the most common OPE techniques that provides unbiased estimates is trajectory based importance sampling (IS). However, due to the high variance of trajectory IS estimates, importance sampling methods based on state-action visitation distributions (SIS) have recently been adopted. Unfortunately, while SIS often provides lower variance estimates for long horizons, estimating the state-action distribution ratios can be challenging and lead to biased estimates. In this paper, we present a new perspective on this bias-variance trade-off and show the existence of a spectrum of estimators whose endpoints are SIS and IS. Additionally, we also establish a spectrum for doubly-robust and weighted version of these estimators. We provide empirical evidence that estimators in this spectrum can be used to trade-off between the bias and variance of IS and SIS and can achieve lower mean-squared error than both IS and SIS.