Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
This provides a formal approach for researchers and practitioners to improve the design of LLM-powered algorithms, though it is incremental as it builds on existing empirical successes.
The authors tackled the challenge of designing and optimizing LLM-based algorithms, which often require extensive trial-and-error, by proposing an analytical framework to systematically analyze how design choices impact accuracy and efficiency, demonstrating its application through case studies with empirical validation.
This work presents an analytical framework for the design and analysis of LLM-based algorithms, i.e., algorithms that contain one or multiple calls of large language models (LLMs) as sub-routines and critically rely on the capabilities of LLMs. While such algorithms, ranging from basic LLM calls with prompt engineering to complicated LLM-powered agentic workflows and compound AI systems, have achieved remarkable empirical success, their design and optimization oftentimes require extensive trial-and-errors and case-by-case analysis. Our proposed framework serves as an attempt to mitigate such headaches, offering a formal and systematic approach for analyzing how the accuracy and efficiency of an LLM-based algorithm will be impacted by critical design choices, such as the pattern and granularity of task decomposition, or the prompt for each LLM call. Through a wide range of case studies covering diverse algorithm patterns (including parallel/hierarchical/recursive task decomposition and generic directed acyclic graphs), we demonstrate the proposed framework in action and derive interesting insights that generalize across scenarios, accompanied by systematic empirical validation in synthetic settings.