Generative Pre-training for Paraphrase Generation by Representing and Predicting Spans in Exemplars
This work offers an incremental improvement in paraphrase generation for NLP practitioners by reducing generic outputs and potentially improving adaptability to new datasets.
This paper addresses the challenge of paraphrase generation, which often suffers from generic outputs or requires retraining for new datasets. The authors propose a GPT-2-based method that uses a novel template masking technique, called first-order masking, to predict spans in masked templates, outperforming competitive baselines in semantic preservation.
Paraphrase generation is a long-standing problem and serves an essential role in many natural language processing problems. Despite some encouraging results, recent methods either confront the problem of favoring generic utterance or need to retrain the model from scratch for each new dataset. This paper presents a novel approach to paraphrasing sentences, extended from the GPT-2 model. We develop a template masking technique, named first-order masking, to masked out irrelevant words in exemplars utilizing POS taggers. So that, the paraphrasing task is changed to predicting spans in masked templates. Our proposed approach outperforms competitive baselines, especially in the semantic preservation aspect. To prevent the model from being biased towards a given template, we introduce a technique, referred to as second-order masking, which utilizes Bernoulli distribution to control the visibility of the first-order-masked template's tokens. Moreover, this technique allows the model to provide various paraphrased sentences in testing by adjusting the second-order-masking level. For scale-up objectives, we compare the performance of two alternatives template-selection methods, which shows that they were equivalent in preserving semantic information.