Wisdom and Delusion of LLM Ensembles for Code Generation and Repair
This addresses the challenge for software engineering practitioners in efficiently leveraging multiple LLMs to improve performance beyond single-model systems, though it is incremental in optimizing ensemble strategies.
The study tackled the problem of maximizing ensemble performance for code generation and repair by comparing ten individual LLMs and three ensembles across three benchmarks, finding that a diversity-based strategy achieves up to 95% of the theoretical potential, which can be 83% above the best single model.
Today's pursuit of a single Large Language Model (LMM) for all software engineering tasks is resource-intensive and overlooks the potential benefits of complementarity, where different models contribute unique strengths. However, the degree to which coding LLMs complement each other and the best strategy for maximizing an ensemble's potential are unclear, leaving practitioners without a clear path to move beyond single-model systems. To address this gap, we empirically compare ten individual LLMs from five families, and three ensembles of these LLMs across three software engineering benchmarks covering code generation and program repair. We assess the complementarity between models and the performance gap between the best individual model and the ensembles. Next, we evaluate various selection heuristics to identify correct solutions from an ensemble's candidate pool. We find that the theoretical upperbound for an ensemble's performance can be 83% above the best single model. Our results show that consensus-based strategies for selecting solutions fall into a "popularity trap," amplifying common but incorrect outputs. In contrast, a diversity-based strategy realizes up to 95% of this theoretical potential, and proves effective even in small two-model ensembles, enabling a cost-efficient way to enhance performance by leveraging multiple LLMs.