People cannot distinguish GPT-4 from a human in a Turing test
This has implications for debates on machine intelligence and suggests AI deception may go undetected, though it is incremental as it builds on prior Turing test research.
The study evaluated GPT-4 in a Turing test and found it was judged as human 54% of the time, outperforming ELIZA (22%) but lagging behind actual humans (67%), providing the first robust empirical demonstration that an AI system passes such a test.
We evaluated 3 systems (ELIZA, GPT-3.5 and GPT-4) in a randomized, controlled, and preregistered Turing test. Human participants had a 5 minute conversation with either a human or an AI, and judged whether or not they thought their interlocutor was human. GPT-4 was judged to be a human 54% of the time, outperforming ELIZA (22%) but lagging behind actual humans (67%). The results provide the first robust empirical demonstration that any artificial system passes an interactive 2-player Turing test. The results have implications for debates around machine intelligence and, more urgently, suggest that deception by current AI systems may go undetected. Analysis of participants' strategies and reasoning suggests that stylistic and socio-emotional factors play a larger role in passing the Turing test than traditional notions of intelligence.