TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networks
This addresses the problem of discrete text generation for NLP applications, but it is incremental as it builds on existing GAN and knowledge distillation techniques.
The paper tackles the challenge of applying GANs to text generation by using knowledge distillation to create continuous sentence representations, achieving better performance in BLEU score and Jensen-Shannon distance compared to traditional GAN-based methods without pre-training.
Text generation is of particular interest in many NLP applications such as machine translation, language modeling, and text summarization. Generative adversarial networks (GANs) achieved a remarkable success in high quality image generation in computer vision,and recently, GANs have gained lots of interest from the NLP community as well. However, achieving similar success in NLP would be more challenging due to the discrete nature of text. In this work, we introduce a method using knowledge distillation to effectively exploit GAN setup for text generation. We demonstrate how autoencoders (AEs) can be used for providing a continuous representation of sentences, which is a smooth representation that assign non-zero probabilities to more than one word. We distill this representation to train the generator to synthesize similar smooth representations. We perform a number of experiments to validate our idea using different datasets and show that our proposed approach yields better performance in terms of the BLEU score and Jensen-Shannon distance (JSD) measure compared to traditional GAN-based text generation approaches without pre-training.