Using Combinatorial Optimization to Design a High quality LLM Solution
This work addresses the challenge of optimizing LLM solutions for researchers and practitioners, but it is incremental as it applies existing combinatorial techniques to a new domain.
The paper tackles the problem of designing high-quality LLM solution pipelines by introducing a combinatorial optimization approach to efficiently explore factors like prompt types and LLM inputs, resulting in a method that reduces manual effort and time consumption in benchmark evaluation.
We introduce a novel LLM based solution design approach that utilizes combinatorial optimization and sampling. Specifically, a set of factors that influence the quality of the solution are identified. They typically include factors that represent prompt types, LLM inputs alternatives, and parameters governing the generation and design alternatives. Identifying the factors that govern the LLM solution quality enables the infusion of subject matter expert knowledge. Next, a set of interactions between the factors are defined and combinatorial optimization is used to create a small subset $P$ that ensures all desired interactions occur in $P$. Each element $p \in P$ is then developed into an appropriate benchmark. Applying the alternative solutions on each combination, $p \in P$ and evaluating the results facilitate the design of a high quality LLM solution pipeline. The approach is especially applicable when the design and evaluation of each benchmark in $P$ is time-consuming and involves manual steps and human evaluation. Given its efficiency the approach can also be used as a baseline to compare and validate an autoML approach that searches over the factors governing the solution.