Does GPT-4 pass the Turing test?
This research addresses the problem of assessing AI's ability to deceive as human in naturalistic communication, with implications for societal consequences, though it is incremental as it builds on prior Turing test evaluations.
The study evaluated GPT-4 in a public online Turing test, finding that its best prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%) but falling short of human participants (66%). Participants based decisions on linguistic style (35%) and socioemotional traits (27%), indicating that intelligence alone is insufficient to pass the test.
We evaluated GPT-4 in a public online Turing test. The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%), but falling short of the baseline set by human participants (66%). Participants' decisions were based mainly on linguistic style (35%) and socioemotional traits (27%), supporting the idea that intelligence, narrowly conceived, is not sufficient to pass the Turing test. Participant knowledge about LLMs and number of games played positively correlated with accuracy in detecting AI, suggesting learning and practice as possible strategies to mitigate deception. Despite known limitations as a test of intelligence, we argue that the Turing test continues to be relevant as an assessment of naturalistic communication and deception. AI models with the ability to masquerade as humans could have widespread societal consequences, and we analyse the effectiveness of different strategies and criteria for judging humanlikeness.