Model-Free and Model-Based Policy Evaluation when Causality is Uncertain
This addresses the challenge of obtaining valid causal estimates in reinforcement learning for decision-makers when causality is uncertain, though it appears incremental in extending existing sensitivity analysis methods.
The paper tackles the problem of off-policy evaluation when unobserved confounders exist, developing worst-case bounds to assess sensitivity to these confounders and showing that model-based approaches with robust MDPs provide sharper lower bounds by leveraging domain knowledge about dynamics.
When decision-makers can directly intervene, policy evaluation algorithms give valid causal estimates. In off-policy evaluation (OPE), there may exist unobserved variables that both impact the dynamics and are used by the unknown behavior policy. These "confounders" will introduce spurious correlations and naive estimates for a new policy will be biased. We develop worst-case bounds to assess sensitivity to these unobserved confounders in finite horizons when confounders are drawn iid each period. We demonstrate that a model-based approach with robust MDPs gives sharper lower bounds by exploiting domain knowledge about the dynamics. Finally, we show that when unobserved confounders are persistent over time, OPE is far more difficult and existing techniques produce extremely conservative bounds.