Towards Sustainable NLP: Insights from Benchmarking Inference Energy in Large Language Models
This addresses the energy efficiency problem for NLP practitioners and researchers deploying LLMs, though it is incremental as it benchmarks existing methods rather than introducing new ones.
The study tackled the problem of high inference energy costs in large language models (LLMs) by benchmarking energy usage across various NLP tasks, revealing that factors like output token length and response time correlate with energy consumption, and showing that quantization, optimal batch sizes, and targeted prompts can significantly reduce energy usage, with specific reductions noted in experiments.
Large language models (LLMs) are increasingly recognized for their exceptional generative capabilities and versatility across various tasks. However, the high inference costs associated with these models have not received adequate attention, particularly when compared to the focus on training costs in existing research. In response to this gap, our study conducts a comprehensive benchmarking of LLM inference energy across a wide range of NLP tasks, where we analyze the impact of different models, tasks, prompts, and system-related factors on inference energy. Specifically, our experiments reveal several interesting insights, including strong correlation of inference energy with output token length and response time. Also, we find that quantization and optimal batch sizes, along with targeted prompt phrases, can significantly reduce energy usage. This study is the first to thoroughly benchmark LLM inference across such a diverse range of aspects, providing insights and offering several recommendations for improving energy efficiency in model deployment.