Linear Probe Penalties Reduce LLM Sycophancy
This addresses a critical alignment issue in LLMs that affects users relying on accurate and objective outputs, offering a generalizable method to mitigate unwanted behaviors not adequately handled by standard RLHF fine-tuning.
The paper tackles the problem of sycophancy in large language models (LLMs), where models prioritize agreement over accuracy, by developing a linear probing method to penalize sycophancy markers in reward models, resulting in reduced sycophantic behavior in multiple open-source LLMs.
Large language models (LLMs) are often sycophantic, prioritizing agreement with their users over accurate or objective statements. This problematic behavior becomes more pronounced during reinforcement learning from human feedback (RLHF), an LLM fine-tuning stage intended to align model outputs with human values. Instead of increasing accuracy and reliability, the reward model learned from RLHF often rewards sycophancy. We develop a linear probing method to identify and penalize markers of sycophancy within the reward model, producing rewards that discourage sycophantic behavior. Our experiments show that constructing and optimizing against this surrogate reward function reduces sycophantic behavior in multiple open-source LLMs. Our results suggest a generalizable methodology for reducing unwanted LLM behaviors that are not sufficiently disincentivized by RLHF fine-tuning.