Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
This work addresses the challenge of ensuring effective collaboration among multiple LLMs for shared goals, which is incremental as it extends existing research on single-model inconsistency to multi-model settings.
The paper tackles the problem of inter-consistency among multiple large language models (LLMs) in collaboration, introducing a formal debate framework (FORD) to examine their ability to reach consensus in commonsense reasoning; experiments show LLMs can collaborate effectively despite inter-inconsistencies, with performance boosted by using GPT-4 as a judge.
Large Language Models (LLMs) have shown impressive capabilities in various applications, but they still face various inconsistency issues. Existing works primarily focus on the inconsistency issues within a single LLM, while we complementarily explore the inter-consistency among multiple LLMs for collaboration. To examine whether LLMs can collaborate effectively to achieve a consensus for a shared goal, we focus on commonsense reasoning, and introduce a formal debate framework (FORD) to conduct a three-stage debate among LLMs with real-world scenarios alignment: fair debate, mismatched debate, and roundtable debate. Through extensive experiments on various datasets, LLMs can effectively collaborate to reach a consensus despite noticeable inter-inconsistencies, but imbalances in their abilities can lead to domination by superior LLMs. Leveraging a more advanced LLM like GPT-4 as an authoritative judge can boost collaboration performance. Our work contributes to understanding the inter-consistency among LLMs and lays the foundation for developing future collaboration methods. Codes and data are available at https://github.com/Waste-Wood/FORD