A Semi-Supervised Text Generation Framework Combining a Deep Transformer and a GAN
This work addresses text generation for NLP applications, but it is incremental as it integrates existing techniques like Transformers and GANs.
The paper tackled the problem of semi-supervised text generation by combining a pre-trained Transformer with a GAN, using Gumbel-Softmax for token discreteness and augmenting real data with GAN samples for fine-tuning, achieving competitive results on benchmark datasets.
This paper introduces a framework that connects a deep generative pre-trained Transformer language model with a generative adversarial network for semi-supervised text generation. In other words, the proposed model is first pre-trained unsupervised on a large and diverse text corpus with 24 layers. Then a simple GAN architecture for synthetic text generation is introduced, and Gumbel-Softmax is applied to handle the discreteness of tokens. The paper also shows a semi-supervised approach where real data is augmented with GAN samples, which is further used to fine-tune the Transformer model on the merged dataset. Detailed theoretical derivations are also included, outlining the proof of the min-max objective function, and an extensive discussion of the Gumbel-Softmax reparameterization trick.