ACtuAL: Actor-Critic Under Adversarial Learning
This addresses a major limitation in GANs for discrete data, such as in reinforcement learning with discrete actions, by enabling training without gradient back-propagation through non-differentiable functions.
The paper tackles the problem of training Generative Adversarial Networks (GANs) with discrete data by reframing the GAN framework to use an actor-critic approach with a Temporal Difference objective, achieving improvements on language modeling tasks over standard Teacher-Forcing methods.
Generative Adversarial Networks (GANs) are a powerful framework for deep generative modeling. Posed as a two-player minimax problem, GANs are typically trained end-to-end on real-valued data and can be used to train a generator of high-dimensional and realistic images. However, a major limitation of GANs is that training relies on passing gradients from the discriminator through the generator via back-propagation. This makes it fundamentally difficult to train GANs with discrete data, as generation in this case typically involves a non-differentiable function. These difficulties extend to the reinforcement learning setting when the action space is composed of discrete decisions. We address these issues by reframing the GAN framework so that the generator is no longer trained using gradients through the discriminator, but is instead trained using a learned critic in the actor-critic framework with a Temporal Difference (TD) objective. This is a natural fit for sequence modeling and we use it to achieve improvements on language modeling tasks over the standard Teacher-Forcing methods.