Scalable Evaluation of Online Facilitation Strategies via Synthetic Simulation of Discussions
This work addresses the high costs of human-involved evaluations for online discussions, offering a scalable synthetic approach for researchers in facilitation and LLM development.
The paper tackles the problem of evaluating online facilitation strategies by proposing a synthetic simulation method using Large Language Models (LLMs) to generate and annotate discussions without human involvement, finding that LLM facilitators improve discussion quality but existing social science strategies do not provide further gains.
Limited large-scale evaluations exist for facilitation strategies of online discussions due to significant costs associated with human involvement. An effective solution is synthetic discussion simulations using Large Language Models (LLMs) to create initial pilot experiments. We propose design principles based on existing methodologies for synthetic discussion generation. Based on these principles, we propose a simple, generalizable, LLM-driven methodology to prototype the development of LLM facilitators by generating synthetic data without human involvement, and which surpasses current baselines. We use our methodology to test whether current Social Science strategies for facilitation can improve the performance of LLM facilitators. We find that, while LLM facilitators significantly improve synthetic discussions, there is no evidence that the application of these strategies leads to further improvements in discussion quality. In an effort to aid research in the field of facilitation, we release a large, publicly available dataset containing LLM-generated and LLM-annotated discussions using multiple open-source models. This dataset can be used for LLM facilitator finetuning as well as behavioral analysis of current out-of-the-box LLMs in the task. We also release an open-source python framework that efficiently implements our methodology at great scale.