Poodle: Seamlessly Scaling Down Large Language Models with Just-in-Time Model Replacement
This addresses cost and energy efficiency for businesses using LLMs, but it is incremental as it builds on existing model search and transfer learning techniques.
The paper tackles the high resource and energy costs of using large language models (LLMs) for simple tasks by proposing just-in-time model replacement (JITR), where LLMs are transparently swapped with cheaper alternatives for recurring tasks, achieving significant savings in cost and energy.
Businesses increasingly rely on large language models (LLMs) to automate simple repetitive tasks instead of developing custom machine learning models. LLMs require few, if any, training examples and can be utilized by users without expertise in model development. However, this comes at the cost of substantially higher resource and energy consumption compared to smaller models, which often achieve similar predictive performance for simple tasks. In this paper, we present our vision for just-in-time model replacement (JITR), where, upon identifying a recurring task in calls to an LLM, the model is replaced transparently with a cheaper alternative that performs well for this specific task. JITR retains the ease of use and low development effort of LLMs, while saving significant cost and energy. We discuss the main challenges in realizing our vision regarding the identification of recurring tasks and the creation of a custom model. Specifically, we argue that model search and transfer learning will play a crucial role in JITR to efficiently identify and fine-tune models for a recurring task. Using our JITR prototype Poodle, we achieve significant savings for exemplary tasks.