OpenHospital: A Thing-in-itself Arena for Evolving and Benchmarking LLM-based Collective Intelligence
This provides a domain-specific tool for researchers in AI and healthcare to benchmark and evolve collective intelligence in LLM agents, though it is incremental as it builds on existing CI concepts.
The authors tackled the lack of a dedicated arena for evolving and benchmarking LLM-based Collective Intelligence by introducing OpenHospital, an interactive environment where physician agents interact with patient agents, resulting in demonstrated effectiveness in fostering and quantifying CI.
Large Language Model (LLM)-based Collective Intelligence (CI) presents a promising approach to overcoming the data wall and continuously boosting the capabilities of LLM agents. However, there is currently no dedicated arena for evolving and benchmarking LLM-based CI. To address this gap, we introduce OpenHospital, an interactive arena where physician agents can evolve CI through interactions with patient agents. This arena employs a data-in-agent-self paradigm that rapidly enhances agent capabilities and provides robust evaluation metrics for benchmarking both medical proficiency and system efficiency. Experiments demonstrate the effectiveness of OpenHospital in both fostering and quantifying CI.