A comparative study of counterfactual estimators
This is an incremental study for reinforcement learning and causal inference researchers, focusing on improving estimator performance in specific settings.
The paper compared off-policy estimators like Empirical Average and Importance Sampling, identifying suboptimal regimes and proposing properties for optimal estimators, showing that fused estimators outperform basic ones in multi-policy scenarios.
We provide a comparative study of several widely used off-policy estimators (Empirical Average, Basic Importance Sampling and Normalized Importance Sampling), detailing the different regimes where they are individually suboptimal. We then exhibit properties optimal estimators should possess. In the case where examples have been gathered using multiple policies, we show that fused estimators dominate basic ones but can still be improved.