PSST: A Benchmark for Evaluation-driven Text Public-Speaking Style Transfer
This addresses the problem of limited style transfer capabilities in AI for complex applications like public speaking, though it is incremental as it builds on existing text style transfer work.
The authors introduced the Public-Speaking Style Transfer (PSST) task to transform official texts into a public-speaking style at the passage level, and found that current large language models struggle with this due to excessive stylization and semantic loss, as shown through comprehensive experiments.
Language style is necessary for AI systems to understand and generate diverse human language accurately. However, previous text style transfer primarily focused on sentence-level data-driven approaches, limiting exploration of potential problems in large language models (LLMs) and the ability to meet complex application needs. To overcome these limitations, we introduce a novel task called Public-Speaking Style Transfer (PSST), which aims to simulate humans to transform passage-level, official texts into a public-speaking style. Grounded in the analysis of real-world data from a linguistic perspective, we decompose public-speaking style into key sub-styles to pose challenges and quantify the style modeling capability of LLMs. For such intricate text style transfer, we further propose a fine-grained evaluation framework to analyze the characteristics and identify the problems of stylized texts. Comprehensive experiments suggest that current LLMs struggle to generate public speaking texts that align with human preferences, primarily due to excessive stylization and loss of semantic information.