Beyond Test-Time Compute Strategies: Advocating Energy-per-Token in LLM Inference
This addresses the problem of high energy consumption in LLM inference for AI practitioners and researchers, offering incremental improvements in efficiency metrics and routing mechanisms.
The paper analyzes the energy-accuracy trade-offs in using test-time compute strategies like Chain-of-Thought prompting with Small Language Models versus larger models, finding that these strategies can reduce computational costs but increase energy usage, and proposes energy-per-token metrics and controlled reasoning to improve sustainability.
Large Language Models (LLMs) demonstrate exceptional performance across diverse tasks but come with substantial energy and computational costs, particularly in request-heavy scenarios. In many real-world applications, the full scale and capabilities of LLMs are often unnecessary, as Small Language Models (SLMs) can provide accurate responses for simpler text generation tasks. When enhanced with advanced reasoning strategies, such as Chain-of-Thought (CoT) prompting or Majority Voting, SLMs can approach the performance of larger models while reducing overall computational requirements. However, these strategies can also introduce additional energy costs, creating an energy-accuracy trade-off. Our analysis examines these trade-offs in test-time compute strategies for smaller models compared to larger ones, using the MMLU benchmark. Additionally, we explore the input-output token dynamics of transformer architectures, which result in nonlinear hardware energy operation curves for LLMs. To bridge AI research with its physical impact, we propose \textit{energy efficiency metrics}, including Energy-per-Token, as complements to traditional accuracy benchmarks. Beyond model selection, we propose controlled reasoning in CoT token generation, using operating curves to regulate reasoning depth dynamically. This vision integrates a energy-aware routing mechanism, ensuring that model selection and inference strategies balance accuracy for sustainable AI deployment.