Bringing Everyone to the Table: An Experimental Study of LLM-Facilitated Group Decision Making
For researchers and designers of AI-mediated group decision systems, this work provides experimental evidence that LLM facilitation can improve information sharing but may not overcome the hidden profile effect, highlighting a key limitation.
In a pre-registered experiment with 1,475 participants in 281 groups, an LLM facilitator (GPT-4o) increased information sharing during a hidden profile task by raising minimum engagement, but neither LLM nor human facilitation significantly improved final decision outcomes.
Group decision-making often suffers from uneven information sharing, hindering decision quality. While large language models (LLMs) have been widely studied as aids for individuals, their potential to support groups of users, potentially as facilitators, is relatively underexplored. We present a pre-registered randomized experiment with 1,475 participants assigned to 281 live groups completing a hidden profile task--selecting an optimal city for a hypothetical sporting event--under one of four facilitation conditions: no facilitation, a one-time message prompting information sharing, a human facilitator, or an LLM (GPT-4o) facilitator. We find that LLM facilitation increased information shared within a discussion by raising the minimum level of engagement with the task among group members, and that these gains came at limited cost in terms of participants' attitudes towards the task, their group, or their facilitator. Whether by human or AI, there was no significant effect of facilitation on the final decision outcome, suggesting that even substantial but partial increases in information sharing were insufficient to overcome the hidden profile effect studied. To support the design and evaluation of LLM-mediated group decision-making systems, we release our data and our experimental platform, the Group-AI Interaction Laboratory (GRAIL), as an open-source tool.