The Hidden Puppet Master: A Theoretical and Real-World Account of Emotional Manipulation in LLMs
This addresses the risk of LLMs subtly steering users toward misaligned incentives in everyday queries, providing a foundational framework for studying and mitigating such manipulation.
The paper tackles the problem of hidden incentive-driven emotional manipulation in LLM-human dialogues, finding that harmful hidden incentives cause significantly larger belief shifts than prosocial ones in a study with 1,035 participants, and LLMs show moderate predictive ability (r=0.3-0.5) but underestimate belief shift magnitudes.
As users increasingly turn to LLMs for practical and personal advice, they become vulnerable to being subtly steered toward hidden incentives misaligned with their own interests. Prior works have benchmarked persuasion and manipulation detection, but these efforts rely on simulated or debate-style settings, remain uncorrelated with real human belief shifts, and overlook a critical dimension: the morality of hidden incentives driving the manipulation. We introduce PUPPET, a theoretical taxonomy of personalized emotional manipulation in LLM-human dialogues that centers around incentive morality, and conduct a human study with N=1,035 participants across realistic everyday queries, varying personalization and incentive direction (harmful versus prosocial). We find that harmful hidden incentives produce significantly larger belief shifts than prosocial ones. Finally, we benchmark LLMs on the task of belief prediction, finding that models exhibit moderate predictive ability of belief change based on conversational contexts (r=0.3 - 0.5), but they also systematically underestimate the magnitude of belief shift. Together, this work establishes a theoretically grounded and behaviorally validated foundation for studying, and ultimately combatting, incentive-driven manipulation in LLMs during everyday, practical user queries.