Collaboration among Multiple Large Language Models for Medical Question Answering
This work addresses medical question answering by improving collaboration among LLMs, but it is incremental as it builds on existing models and datasets.
The authors tackled the problem of insufficient synergy among multiple large language models (LLMs) in medical question answering by proposing a multi-LLM collaboration framework, which boosted reasoning ability and alleviated divergence among models, with observed concurrence between confidence and accuracy.
Empowered by vast internal knowledge reservoir, the new generation of large language models (LLMs) demonstrate untapped potential to tackle medical tasks. However, there is insufficient effort made towards summoning up a synergic effect from multiple LLMs' expertise and background. In this study, we propose a multi-LLM collaboration framework tailored on a medical multiple-choice questions dataset. Through post-hoc analysis on 3 pre-trained LLM participants, our framework is proved to boost all LLMs reasoning ability as well as alleviate their divergence among questions. We also measure an LLM's confidence when it confronts with adversary opinions from other LLMs and observe a concurrence between LLM's confidence and prediction accuracy.