Understanding the Effects of RLHF on the Quality and Detectability of LLM-Generated Texts
This addresses the problem of detecting AI-generated text for security and integrity, but it is incremental as it builds on existing RLHF and detection methods.
The study investigated how Reinforcement Learning from Human Feedback (RLHF) affects the quality and detectability of LLM-generated texts, finding that RLHF improves quality but makes outputs more detectable, lengthy, and repetitive, with training-based detectors being vulnerable to short or code-mixed texts while zero-shot detectors are more robust.
Large Language Models (LLMs) have demonstrated exceptional performance on a range of downstream NLP tasks by generating text that closely resembles human writing. However, the ease of achieving this similarity raises concerns from potential malicious uses at scale by bad actors, as LLM-generated text becomes increasingly difficult to discern from human text. Although detection methods have been developed to address this issue, bad actors can further manipulate LLM-generated texts to make them less detectable. In this work, we study how further editing texts with Reinforcement Learning from Human Feedback (RLHF), which aligns model outputs with human preferences, affects (a) the quality of generated texts for two tasks, and (b) the performance of LLM-generated text detectors, looking at both training-based and zero-shot detection methods. Although RLHF improves the quality of LLM-generated texts, we find that it also tends to produce more detectable, lengthy, and repetitive outputs. Additionally, we observe that training-based detectors are vulnerable to short texts and to texts that incorporate code, whereas zero-shot detectors exhibit greater robustness.