Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
This work addresses the problem of efficient LLM deployment across diverse hardware platforms for developers and system designers, though it is incremental as it builds on existing analytical modeling approaches.
The authors tackled the challenge of forecasting LLM inference performance on heterogeneous hardware by introducing LIFE, a lightweight analytical framework that predicts metrics like time-to-first-token and tokens-per-second using only hardware specifications, achieving validation across CPUs, NPUs, and GPUs without extensive benchmarking.
Large language models (LLMs) have been increasingly deployed as local agents on personal devices with CPUs, NPUs and integrated GPUs. However, forecasting inference performance on devices with such heterogeneity remains challenging due to the dynamic compute and memory demands. Existing approaches rely on GPU benchmarking or machine learning-based latency predictors, which are often hardware-specific and lack generalizability. To this end, we introduce LIFE, a lightweight and modular analytical framework that is comprised of modular analytical model of operators, configurable to characterize LLM inference workloads in a hardware and dataset-agnostic manner. LIFE characterizes the influence of software and model optimizations, such as quantization, KV cache compression, LoRA adapters, chunked prefill, different attentions, and operator fusion, on performance metrics such as time-to-first-token (TTFT), time-per-output-token (TPOT) and tokens-per-second (TPS). LIFE enables performance forecasting using only hardware specifications, such as TOPS and memory bandwidth, without requiring extensive dataset benchmarking. We validate LIFE's forecasting with inference on AMD Ryzen CPUs, NPUs, iGPUs and NVIDIA V100 GPUs, with Llama2-7B variants, demonstrating the utility of LIFE in forecasting LLM performance through lens of system efficiency to enable efficient LLM deployment across different hardware platforms.