LLM Chemistry Estimation for Multi-LLM Recommendation
This addresses the need for better ensemble selection in multi-LLM systems, offering a diagnostic tool for researchers and practitioners, though it appears incremental as it builds on existing collaboration approaches.
The paper tackles the problem of multi-LLM collaboration by introducing LLM Chemistry, a framework that measures synergistic or antagonistic behaviors in model combinations, with evaluation on classification, summarization, and program repair tasks providing initial evidence for task-dependent effects.
Multi-LLM collaboration promises accurate, robust, and context-aware solutions, yet existing approaches rely on implicit selection and output assessment without analyzing whether collaborating models truly complement or conflict. We introduce LLM Chemistry -- a framework that measures when LLM combinations exhibit synergistic or antagonistic behaviors that shape collective performance beyond individual capabilities. We formalize the notion of chemistry among LLMs, propose algorithms that quantify it by analyzing interaction dependencies, and recommend optimal model ensembles accordingly. Our theoretical analysis shows that chemistry among collaborating LLMs is most evident under heterogeneous model profiles, with its outcome impact shaped by task type, group size, and complexity. Evaluation on classification, summarization, and program repair tasks provides initial evidence for these task-dependent effects, thereby reinforcing our theoretical results. This establishes LLM Chemistry as both a diagnostic factor in multi-LLM systems and a foundation for ensemble recommendation.