Opportunities and Risks of LLMs for Scalable Deliberation with Polis
This work addresses the problem of scaling deliberative processes for public engagement, though it is incremental as it builds on existing Polis technology with LLM integration.
The paper explores using Large Language Models (LLMs) like Claude to enhance the Polis platform for scalable deliberation, finding that LLMs can efficiently augment human intelligence in facilitating and summarizing conversations, with summarization enabling new methods for collective meaning-making, but noting that LLM context limitations significantly impact insight and quality.
Polis is a platform that leverages machine intelligence to scale up deliberative processes. In this paper, we explore the opportunities and risks associated with applying Large Language Models (LLMs) towards challenges with facilitating, moderating and summarizing the results of Polis engagements. In particular, we demonstrate with pilot experiments using Anthropic's Claude that LLMs can indeed augment human intelligence to help more efficiently run Polis conversations. In particular, we find that summarization capabilities enable categorically new methods with immense promise to empower the public in collective meaning-making exercises. And notably, LLM context limitations have a significant impact on insight and quality of these results. However, these opportunities come with risks. We discuss some of these risks, as well as principles and techniques for characterizing and mitigating them, and the implications for other deliberative or political systems that may employ LLMs. Finally, we conclude with several open future research directions for augmenting tools like Polis with LLMs.