Confounded Causal Imitation Learning with Instrumental Variables
This addresses the issue of biased policy estimation in imitation learning for researchers and practitioners, though it is incremental as it builds on existing IV methods.
The paper tackles the problem of imitation learning from demonstrations being biased by unmeasured confounders, and proposes a Confounded Causal Imitation Learning (C2L) model using instrumental variables (IV) to break this confounding gap, with extensive experiments verifying its effectiveness in identifying valid IVs and learning policies.
Imitation learning from demonstrations usually suffers from the confounding effects of unmeasured variables (i.e., unmeasured confounders) on the states and actions. If ignoring them, a biased estimation of the policy would be entailed. To break up this confounding gap, in this paper, we take the best of the strong power of instrumental variables (IV) and propose a Confounded Causal Imitation Learning (C2L) model. This model accommodates confounders that influence actions across multiple timesteps, rather than being restricted to immediate temporal dependencies. We develop a two-stage imitation learning framework for valid IV identification and policy optimization. In particular, in the first stage, we construct a testing criterion based on the defined pseudo-variable, with which we achieve identifying a valid IV for the C2L models. Such a criterion entails the sufficient and necessary identifiability conditions for IV validity. In the second stage, with the identified IV, we propose two candidate policy learning approaches: one is based on a simulator, while the other is offline. Extensive experiments verified the effectiveness of identifying the valid IV as well as learning the policy.