Follow the Neurally-Perturbed Leader for Adversarial Training
This addresses convergence issues in adversarial learning frameworks, offering a more stable training method for applications like imitation learning, though it appears incremental in improving existing game-theoretic models.
The paper tackles the problem of weak convergence and cyclic dynamics in zero-sum adversarial training by proposing a novel follow the leader algorithm with neural perturbations, which guarantees convergence to mixed Nash equilibrium and is validated on convex, non-convex, and generative adversarial architectures.
Game-theoretic models of learning are a powerful set of models that optimize multi-objective architectures. Among these models are zero-sum architectures that have inspired adversarial learning frameworks. An important shortcoming of these zeros-sum architectures is that gradient-based training leads to weak convergence and cyclic dynamics. We propose a novel follow the leader training algorithm for zeros-sum architectures that guarantees convergence to mixed Nash equilibrium without cyclic behaviors. It is a special type of follow the perturbed leader algorithm where perturbations are the result of a neural mediating agent. We validate our theoretical results by applying this training algorithm to games with convex and non-convex loss as well as generative adversarial architectures. Moreover, we customize the implementation of this algorithm for adversarial imitation learning applications. At every step of the training, the mediator agent perturbs the observations with generated codes. As a result of these mediating codes, the proposed algorithm is also efficient for learning in environments with various factors of variations. We validate our assertion by using a procedurally generated game environment as well as synthetic data. Github implementation is available.