Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training
This addresses style transfer limitations of large models for applications needing privacy and low-cost deployment, though it appears incremental in method.
The paper tackles complex text style transfer by creating a new dataset and training small language models with latent attribute pre-training, achieving state-of-the-art performance for few-shot style transfer.
In this work, we introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with 700 rephrased sentences and 1,000 sentences from the game Genshin Impact. While large language models (LLM) have shown promise in complex text style transfer, they have drawbacks such as data privacy concerns, network instability, and high deployment costs. To address these issues, we explore the effectiveness of small models (less than T5-3B) with implicit style pre-training through contrastive learning. We also propose a method for automated evaluation of text generation quality based on alignment with human evaluations using ChatGPT. Finally, we compare our approach with existing methods and show that our model achieves state-of-art performances of few-shot text style transfer models.