Statistical Estimation of Confounded Linear MDPs: An Instrumental Variable Approach
This addresses the challenge of accurate policy evaluation in reinforcement learning when data is confounded, which is critical for applications like healthcare and robotics, though it is an incremental advancement building on existing instrumental variable methods.
The paper tackles the problem of off-policy evaluation in Markov decision processes with unobservable confounders by proposing a two-stage estimator using instrumental variables, achieving a non-asymptotic error bound of O(n^{-1/2}) and asymptotic normality with a rate of n^{1/2}.
In an Markov decision process (MDP), unobservable confounders may exist and have impacts on the data generating process, so that the classic off-policy evaluation (OPE) estimators may fail to identify the true value function of the target policy. In this paper, we study the statistical properties of OPE in confounded MDPs with observable instrumental variables. Specifically, we propose a two-stage estimator based on the instrumental variables and establish its statistical properties in the confounded MDPs with a linear structure. For non-asymptotic analysis, we prove a $\mathcal{O}(n^{-1/2})$-error bound where $n$ is the number of samples. For asymptotic analysis, we prove that the two-stage estimator is asymptotically normal with a typical rate of $n^{1/2}$. To the best of our knowledge, we are the first to show such statistical results of the two-stage estimator for confounded linear MDPs via instrumental variables.