Are they human? Detecting large language models by probing human memory constraints
This addresses the threat to online research validity from LLMs posing as humans, offering a detection method based on cognitive science.
The paper tackles the problem of detecting large language models (LLMs) in online behavioral research by using tasks that exploit human working memory constraints, showing that cognitive modeling on a serial recall task can distinguish LLMs from humans even when LLMs mimic these constraints.
The validity of online behavioral research relies on study participants being human rather than machine. In the past, it was possible to detect machines by posing simple challenges that were easily solved by humans but not by machines. General-purpose agents based on large language models (LLMs) can now solve many of these challenges, threatening the validity of online behavioral research. Here we explore the idea of detecting humanness by using tasks that machines can solve too well to be human. Specifically, we probe for the existence of an established human cognitive constraint: limited working memory capacity. We show that cognitive modeling on a standard serial recall task can be used to distinguish online participants from LLMs even when the latter are specifically instructed to mimic human working memory constraints. Our results demonstrate that it is viable to use well-established cognitive phenomena to distinguish LLMs from humans.