The Costly Dilemma: Generalization, Evaluation and Cost-Optimal Deployment of Large Language Models
This work provides a practical framework for enterprises to manage deployment trade-offs, but it is incremental as it adapts existing concepts to large language models.
The paper addresses the challenge of balancing generalization, evaluation, and cost-optimality in deploying large language models, proposing a tailored framework to help enterprises assess these factors before investment.
When deploying machine learning models in production for any product/application, there are three properties that are commonly desired. First, the models should be generalizable, in that we can extend it to further use cases as our knowledge of the domain area develops. Second they should be evaluable, so that there are clear metrics for performance and the calculation of those metrics in production settings are feasible. Finally, the deployment should be cost-optimal as far as possible. In this paper we propose that these three objectives (i.e. generalization, evaluation and cost-optimality) can often be relatively orthogonal and that for large language models, despite their performance over conventional NLP models, enterprises need to carefully assess all the three factors before making substantial investments in this technology. We propose a framework for generalization, evaluation and cost-modeling specifically tailored to large language models, offering insights into the intricacies of development, deployment and management for these large language models.