On Value Discrepancy of Imitation Learning
This provides theoretical insights for imitation learning researchers, though it's incremental as it analyzes existing methods rather than proposing new ones.
The paper tackles the theoretical analysis of imitation learning approaches by proposing a framework based on discrepancy propagation analysis, showing that GAIL has value discrepancy of O((1-γ)^{-1}) while behavioral cloning has O((1-γ)^{-2}), indicating GAIL has less compounding errors.
Imitation learning trains a policy from expert demonstrations. Imitation learning approaches have been designed from various principles, such as behavioral cloning via supervised learning, apprenticeship learning via inverse reinforcement learning, and GAIL via generative adversarial learning. In this paper, we propose a framework to analyze the theoretical property of imitation learning approaches based on discrepancy propagation analysis. Under the infinite-horizon setting, the framework leads to the value discrepancy of behavioral cloning in an order of O((1-γ)^{-2}). We also show that the framework leads to the value discrepancy of GAIL in an order of O((1-γ)^{-1}). It implies that GAIL has less compounding errors than behavioral cloning, which is also verified empirically in this paper. To the best of our knowledge, we are the first one to analyze GAIL's performance theoretically. The above results indicate that the proposed framework is a general tool to analyze imitation learning approaches. We hope our theoretical results can provide insights for future improvements in imitation learning algorithms.