CLEAR: A Comprehensive Linguistic Evaluation of Argument Rewriting by Large Language Models
This work addresses the need for comprehensive evaluation of LLMs in text rewriting tasks for researchers and practitioners, but it is incremental as it applies existing metrics to a new domain.
The paper tackles the problem of evaluating how large language models (LLMs) rewrite argumentative texts, specifically in Argument Improvement (ArgImp), by introducing CLEAR, an evaluation pipeline with 57 metrics across four linguistic levels. The result shows that models shorten texts, increase average word length, merge sentences, and improve persuasion and coherence.
While LLMs have been extensively studied on general text generation tasks, there is less research on text rewriting, a task related to general text generation, and particularly on the behavior of models on this task. In this paper we analyze what changes LLMs make in a text rewriting setting. We focus specifically on argumentative texts and their improvement, a task named Argument Improvement (ArgImp). We present CLEAR: an evaluation pipeline consisting of 57 metrics mapped to four linguistic levels: lexical, syntactic, semantic and pragmatic. This pipeline is used to examine the qualities of LLM-rewritten arguments on a broad set of argumentation corpora and compare the behavior of different LLMs on this task and analyze the behavior of different LLMs on this task in terms of linguistic levels. By taking all four linguistic levels into consideration, we find that the models perform ArgImp by shortening the texts while simultaneously increasing average word length and merging sentences. Overall we note an increase in the persuasion and coherence dimensions.