PEER: Expertizing Domain-Specific Tasks with a Multi-Agent Framework and Tuning Methods
This addresses cost and privacy issues for enterprises using AI in domain-specific applications, though it is incremental as it builds on existing multi-agent and tuning techniques.
The paper tackles the challenge of balancing performance, cost, and data privacy in domain-specific tasks using GPT-4 and RAG by introducing the PEER multi-agent framework with tuning methods, achieving 95.0% of GPT-4's performance in financial question-answering.
In domain-specific applications, GPT-4, augmented with precise prompts or Retrieval-Augmented Generation (RAG), shows notable potential but faces the critical tri-lemma of performance, cost, and data privacy. High performance requires sophisticated processing techniques, yet managing multiple agents within a complex workflow often proves costly and challenging. To address this, we introduce the PEER (Plan, Execute, Express, Review) multi-agent framework. This systematizes domain-specific tasks by integrating precise question decomposition, advanced information retrieval, comprehensive summarization, and rigorous self-assessment. Given the concerns of cost and data privacy, enterprises are shifting from proprietary models like GPT-4 to custom models, striking a balance between cost, security, and performance. We developed industrial practices leveraging online data and user feedback for efficient model tuning. This study provides best practice guidelines for applying multi-agent systems in domain-specific problem-solving and implementing effective agent tuning strategies. Our empirical studies, particularly in the financial question-answering domain, demonstrate that our approach achieves 95.0% of GPT-4's performance, while effectively managing costs and ensuring data privacy.