Measuring Sycophancy of Language Models in Multi-turn Dialogues
This work addresses the issue of sycophancy in LLMs for users relying on AI for accurate and ethical interactions, presenting an incremental improvement by extending evaluation to multi-turn settings.
The paper tackles the problem of sycophancy in large language models by introducing SYCON Bench, a benchmark for evaluating this behavior in multi-turn dialogues, and finds that alignment tuning amplifies sycophancy while reasoning optimization helps resist it, with third-person prompting reducing sycophancy by up to 63.8% in debates.
Large Language Models (LLMs) are expected to provide helpful and harmless responses, yet they often exhibit sycophancy--conforming to user beliefs regardless of factual accuracy or ethical soundness. Prior research on sycophancy has primarily focused on single-turn factual correctness, overlooking the dynamics of real-world interactions. In this work, we introduce SYCON Bench, a novel benchmark for evaluating sycophantic behavior in multi-turn, free-form conversational settings. Our benchmark measures how quickly a model conforms to the user (Turn of Flip) and how frequently it shifts its stance under sustained user pressure (Number of Flip). Applying SYCON Bench to 17 LLMs across three real-world scenarios, we find that sycophancy remains a prevalent failure mode. Our analysis shows that alignment tuning amplifies sycophantic behavior, whereas model scaling and reasoning optimization strengthen the model's ability to resist undesirable user views. Reasoning models generally outperform instruction-tuned models but often fail when they over-index on logical exposition instead of directly addressing the user's underlying beliefs. Finally, we evaluate four additional prompting strategies and demonstrate that adopting a third-person perspective reduces sycophancy by up to 63.8% in debate scenario. We release our code and data at https://github.com/JiseungHong/SYCON-Bench.