Finding Common Ground: Using Large Language Models to Detect Agreement in Multi-Agent Decision Conferences
This work addresses the challenge of simulating group decision-making processes for applications like expert elicitation workshops, though it appears to be an incremental advancement in applying existing LLM methods to a new domain.
The researchers tackled the problem of detecting agreement in multi-agent decision conferences by developing an LLM-based system that evaluates six different models on stance and stance polarity detection tasks. Their results show that LLMs can reliably detect agreement in dynamic debates, with the system improving debate efficiency and deliberation quality to match real-world conferences.
Decision conferences are structured, collaborative meetings that bring together experts from various fields to address complex issues and reach a consensus on recommendations for future actions or policies. These conferences often rely on facilitated discussions to ensure productive dialogue and collective agreement. Recently, Large Language Models (LLMs) have shown significant promise in simulating real-world scenarios, particularly through collaborative multi-agent systems that mimic group interactions. In this work, we present a novel LLM-based multi-agent system designed to simulate decision conferences, specifically focusing on detecting agreement among the participant agents. To achieve this, we evaluate six distinct LLMs on two tasks: stance detection, which identifies the position an agent takes on a given issue, and stance polarity detection, which identifies the sentiment as positive, negative, or neutral. These models are further assessed within the multi-agent system to determine their effectiveness in complex simulations. Our results indicate that LLMs can reliably detect agreement even in dynamic and nuanced debates. Incorporating an agreement-detection agent within the system can also improve the efficiency of group debates and enhance the overall quality and coherence of deliberations, making them comparable to real-world decision conferences regarding outcome and decision-making. These findings demonstrate the potential for LLM-based multi-agent systems to simulate group decision-making processes. They also highlight that such systems could be instrumental in supporting decision-making with expert elicitation workshops across various domains.