GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution
This addresses a specific technical bottleneck in sequence generation for researchers in generative modeling, but appears incremental as it applies an existing method (Gumbel-softmax) to a known problem.
The paper tackled the problem of generating sequences of discrete elements with GANs, which face limitations due to non-differentiability, by using the Gumbel-softmax distribution as a continuous approximation, and evaluated its performance in this task.
Generative Adversarial Networks (GAN) have limitations when the goal is to generate sequences of discrete elements. The reason for this is that samples from a distribution on discrete objects such as the multinomial are not differentiable with respect to the distribution parameters. This problem can be avoided by using the Gumbel-softmax distribution, which is a continuous approximation to a multinomial distribution parameterized in terms of the softmax function. In this work, we evaluate the performance of GANs based on recurrent neural networks with Gumbel-softmax output distributions in the task of generating sequences of discrete elements.