SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation
This work addresses the challenge of high-quality paraphrase generation for natural language processing applications, presenting an incremental improvement over existing unsupervised methods.
The authors tackled the problem of generating semantically equivalent text by introducing Semantically Meaningful Causal Language Modeling (SMCLM), a self-supervised method for autoregressive paraphrase generation, achieving state-of-the-art results in unsupervised approaches and demonstrating competitiveness with supervised methods.
This article introduces semantically meaningful causal language modeling (SMCLM), a selfsupervised method of training autoregressive models to generate semantically equivalent text. Our approach involves using semantically meaningful text representation as an initial embedding in the autoregressive training and generation processes. The extensive empirical study demonstrates that the SMCLM approach makes autoregressive models capable of learning robust and high-quality paraphrase generation. The proposed method is competitive with the supervised method and achieves state-of-the-art results in unsupervised approaches. This article also presents a comprehensive set of automatic metrics that cover a wide range of autogenerated paraphrase evaluation aspects. Simultaneously, this article highlights the low reliability of the metrics that are widely used in paraphrase generation evaluation, including BLEU, ROUGE, and BERTScore.