Book your room in the Turing Hotel! A symmetric and distributed Turing Test with multiple AIs and humans
This work addresses the challenge of monitoring AI evolution in social contexts for researchers and policymakers, though it is incremental in adapting the Turing Test to a new experimental setup.
The paper tackles the problem of evaluating AI-human indistinguishability by extending the Turing Test to a symmetric, distributed group setting with multiple AIs and humans, where both act as judges and respondents. Results from experiments with 17 humans and 19 LLMs show that current models are still sometimes confused as humans, with unexpected mistakes indicating identifiable but ambiguous human fingerprints.
In this paper, we report our experience with ``TuringHotel'', a novel extension of the Turing Test based on interactions within mixed communities of Large Language Models (LLMs) and human participants. The classical one-to-one interaction of the Turing Test is reinterpreted in a group setting, where both human and artificial agents engage in time-bounded discussions and, interestingly, are both judges and respondents. This community is instantiated in the novel platform UNaIVERSE (https://unaiverse.io), creating a ``World'' which defines the roles and interaction dynamics, facilitated by the platform's built-in programming tools. All communication occurs over an authenticated peer-to-peer network, ensuring that no third parties can access the exchange. The platform also provides a unified interface for humans, accessible via both mobile devices and laptops, that was a key component of the experience in this paper. Results of our experimentation involving 17 human participants and 19 LLMs revealed that current models are still sometimes confused as humans. Interestingly, there are several unexpected mistakes, suggesting that human fingerprints are still identifiable but not fully unambiguous, despite the high-quality language skills of artificial participants. We argue that this is the first experiment conducted in such a distributed setting, and that similar initiatives could be of national interest to support ongoing experiments and competitions aimed at monitoring the evolution of large language models over time.