Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective
It addresses training instability in imitation learning, an incremental improvement for researchers and practitioners using GAIL methods.
This paper investigates gradient explosion in Generative Adversarial Imitation Learning (GAIL), finding that deterministic policy GAIL (DE-GAIL) suffers from instability and divergence due to large expert-imitator policy disparities, while stochastic policy GAIL (ST-GAIL) converges reliably, and proposes CREDO, a reward-clipping strategy that improves data efficiency and stability.
Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with stochastic policy (ST-GAIL). We begin with the observation that the training can be highly unstable for DE-GAIL at the beginning of the training phase and end up divergence. Conversely, the ST-GAIL training trajectory remains consistent, reliably converging. To shed light on these disparities, we provide an explanation from a theoretical perspective. By establishing a probabilistic lower bound for GAIL, we demonstrate that gradient explosion is an inevitable outcome for DE-GAIL due to occasionally large expert-imitator policy disparity, whereas ST-GAIL does not have the issue with it. To substantiate our assertion, we illustrate how modifications in the reward function can mitigate the gradient explosion challenge. Finally, we propose CREDO, a simple yet effective strategy that clips the reward function during the training phase, allowing the GAIL to enjoy high data efficiency and stable trainability.