Persuade Me if You Can: A Framework for Evaluating Persuasion Effectiveness and Susceptibility Among Large Language Models
This work addresses the need for scalable evaluation of persuasion risks in LLMs to develop safer AI systems, though it is incremental as it builds on existing multi-agent interaction methods.
The paper tackles the problem of evaluating persuasion capabilities and susceptibility in Large Language Models (LLMs) by introducing the Persuade Me If You Can (PMIYC) framework, which uses multi-agent interactions to measure these traits across diverse models. Results show that Llama-3.3-70B and GPT-4o have similar persuasive effectiveness, outperforming Claude 3 Haiku by 30%, while GPT-4o demonstrates over 50% greater resistance to misinformation persuasion compared to Llama-3.3-70B.
Large Language Models (LLMs) demonstrate persuasive capabilities that rival human-level persuasion. While these capabilities can be used for social good, they also present risks of potential misuse. Moreover, LLMs' susceptibility to persuasion raises concerns about alignment with ethical principles. To study these dynamics, we introduce Persuade Me If You Can (PMIYC), an automated framework for evaluating persuasion through multi-agent interactions. Here, Persuader agents engage in multi-turn conversations with the Persuadee agents, allowing us to measure LLMs' persuasive effectiveness and their susceptibility to persuasion. We conduct comprehensive evaluations across diverse LLMs, ensuring each model is assessed against others in both subjective and misinformation contexts. We validate the efficacy of our framework through human evaluations and show alignment with prior work. PMIYC offers a scalable alternative to human annotation for studying persuasion in LLMs. Through PMIYC, we find that Llama-3.3-70B and GPT-4o exhibit similar persuasive effectiveness, outperforming Claude 3 Haiku by 30%. However, GPT-4o demonstrates over 50% greater resistance to persuasion for misinformation compared to Llama-3.3-70B. These findings provide empirical insights into the persuasive dynamics of LLMs and contribute to the development of safer AI systems.