Deliberate Lab: A Platform for Real-Time Human-AI Social Experiments
This platform addresses the need for accessible and replicable tools for social and behavioral scientists studying human-AI collaboration, though it is incremental as it builds on existing concepts with standardized support.
The authors tackled the problem of underdeveloped experimental infrastructure for studying human-AI interactions by developing Deliberate Lab, an open-source platform that supports real-time, multi-party experiments with both human and LLM-based agents, reporting on a 12-month deployment with 88 experimenters and 9,195 participants.
Social and behavioral scientists increasingly aim to study how humans interact, collaborate, and make decisions alongside artificial intelligence. However, the experimental infrastructure for such work remains underdeveloped: (1) few platforms support real-time, multi-party studies at scale; (2) most deployments require bespoke engineering, limiting replicability and accessibility, and (3) existing tools do not treat AI agents as first-class participants. We present Deliberate Lab, an open-source platform for large-scale, real-time behavioral experiments that supports both human participants and large language model (LLM)-based agents. We report on a 12-month public deployment of the platform (N=88 experimenters, N=9195 experiment participants), analyzing usage patterns and workflows. Case studies and usage scenarios are aggregated from platform users, complemented by in-depth interviews with select experimenters. By lowering technical barriers and standardizing support for hybrid human-AI experimentation, Deliberate Lab expands the methodological repertoire for studying collective decision-making and human-centered AI.