Teaching LLMs Human-Like Editing of Inappropriate Argumentation via Reinforcement Learning
For users of LLMs who need text editing that preserves meaning while improving appropriateness, this method offers more human-like edits.
LLMs often edit arguments in scattered, meaning-changing ways unlike humans. The authors use reinforcement learning to teach LLMs human-like, self-contained edits, achieving appropriateness close to full rewriting and outperforming baselines.
Editing human-written text has become a standard use case of large language models (LLMs), for example, to make one's arguments more appropriate for a discussion. Comparing human to LLM-generated edits, however, we observe a mismatch in editing strategies: While LLMs often perform multiple scattered edits and tend to change meaning notably, humans rather encapsulate dependent changes in self-contained, meaning-preserving edits. In this paper, we present a reinforcement learning approach that teaches LLMs human-like editing to improve the appropriateness of arguments. Our approach produces self-contained sentence-level edit suggestions that can be accepted or rejected independently. We train the approach using group relative policy optimization with a multi-component reward function that jointly optimizes edit-level semantic similarity, fluency, and pattern conformity as well as argument-level appropriateness. In automatic and human evaluation, it outperforms competitive baselines and the state of the art in human-like editing, with multi-round editing achieving appropriateness close to full rewriting.