Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task
This work addresses the problem of using LLMs as substitutes for human participants in cognitive tasks, revealing key limitations for researchers in cognitive science and AI.
The study investigated whether large language models (LLMs) can simulate human behavioral variability in the phonemic fluency task, finding that while some configurations matched human averages, none reproduced the full scope of human variability, with LLM outputs being less diverse and structurally rigid.
Large language models (LLMs) are increasingly explored as substitutes for human participants in cognitive tasks, but their ability to simulate human behavioral variability remains unclear. This study examines whether LLMs can approximate individual differences in the phonemic fluency task, where participants generate words beginning with a target letter. We evaluated 34 model configurations, varying prompt specificity, sampling temperature, and model type, and compared outputs to responses from 106 human participants. While some configurations, especially Claude 3.7 Sonnet, matched human averages and lexical preferences, none reproduced the scope of human variability. LLM outputs were consistently less diverse and structurally rigid, and LLM ensembles failed to increase diversity. Network analyses further revealed fundamental differences in retrieval structure between humans and models. These results highlight key limitations in using LLMs to simulate human cognition and behavior.