Not Yet AlphaFold for the Mind: Evaluating Centaur as a Synthetic Participant
This addresses the need for a reliable participant simulator in behavioral sciences to accelerate hypothesis testing, but the work is incremental as it critiques an existing model rather than proposing a new solution.
The paper evaluated Centaur, an LLM fine-tuned on human data, as a participant simulator for cognitive tasks, finding that while it has strong predictive accuracy, its generative behavior systematically diverges from human data, indicating it does not yet meet the standards of a reliable simulator.
Simulators have revolutionized scientific practice across the natural sciences. By generating data that reliably approximate real-world phenomena, they enable scientists to accelerate hypothesis testing and optimize experimental designs. This is perhaps best illustrated by AlphaFold, a Nobel-prize winning simulator in chemistry that predicts protein structures from amino acid sequences, enabling rapid prototyping of molecular interactions, drug targets, and protein functions. In the behavioral sciences, a reliable participant simulator - a system capable of producing human-like behavior across cognitive tasks - would represent a similarly transformative advance. Recently, Binz et al. introduced Centaur, a large language model (LLM) fine-tuned on human data from 160 experiments, proposing its use not only as a model of cognition but also as a participant simulator for "in silico prototyping of experimental studies", e.g., to advance automated cognitive science. Here, we review the core criteria for a participant simulator and assess how well Centaur meets them. Although Centaur demonstrates strong predictive accuracy, its generative behavior - a critical criterion for a participant simulator - systematically diverges from human data. This suggests that, while Centaur is a significant step toward predicting human behavior, it does not yet meet the standards of a reliable participant simulator or an accurate model of cognition.