LLMs Among Us: Generative AI Participating in Digital Discourse
This addresses the threat of AI-driven misinformation and deception in digital discourse, though it is incremental in evaluating existing LLMs.
The study tackled the problem of LLMs impersonating humans in social media by developing an experimental framework on Mastodon, finding that participants could only correctly identify bots vs. humans 42% of the time, with persona choice having more impact than the LLM model used.
The emergence of Large Language Models (LLMs) has great potential to reshape the landscape of many social media platforms. While this can bring promising opportunities, it also raises many threats, such as biases and privacy concerns, and may contribute to the spread of propaganda by malicious actors. We developed the "LLMs Among Us" experimental framework on top of the Mastodon social media platform for bot and human participants to communicate without knowing the ratio or nature of bot and human participants. We built 10 personas with three different LLMs, GPT-4, LLama 2 Chat, and Claude. We conducted three rounds of the experiment and surveyed participants after each round to measure the ability of LLMs to pose as human participants without human detection. We found that participants correctly identified the nature of other users in the experiment only 42% of the time despite knowing the presence of both bots and humans. We also found that the choice of persona had substantially more impact on human perception than the choice of mainstream LLMs.