A bi-objective $ε$-constrained framework for quality-cost optimization in language model ensembles
This addresses cost efficiency for users of open-sourced LLM ensembles, but appears incremental as it builds on existing ensembling methods with a new optimization constraint.
The paper tackles the problem of balancing response quality and cost efficiency in language model ensembles by formulating a bi-objective optimization with a budget constraint, reducing it to a knapsack problem. It empirically shows the framework outperforms existing ensembling approaches in quality while significantly reducing costs, though no specific numbers are provided.
We propose an ensembling framework that uses diverse open-sourced Large Language Models (LLMs) to achieve high response quality while maintaining cost efficiency. We formulate a bi-objective optimization problem to represent the quality-cost tradeoff and then introduce an additional budget constraint that reduces the problem to a straightforward 0/1 knapsack problem. We empirically demonstrate that our framework outperforms the existing ensembling approaches in response quality while significantly reducing costs.