Memory Access Characterization of Large Language Models in CPU Environment and its Potential Impacts
This work addresses the challenge of running LLMs in resource-constrained environments like those with energy or cost limitations, but it is incremental as it focuses on optimizing existing hardware rather than introducing new methods.
The paper tackled the problem of slow large language model inference on CPUs by analyzing memory access patterns and cache configurations, finding that specific cache modifications could improve performance, though concrete numbers were not provided.
As machine learning algorithms are shown to be an increasingly valuable tool, the demand for their access has grown accordingly. Oftentimes, it is infeasible to run inference with larger models without an accelerator, which may be unavailable in environments that have constraints such as energy consumption, security, or cost. To increase the availability of these models, we aim to improve the LLM inference speed on a CPU-only environment by modifying the cache architecture. To determine what improvements could be made, we conducted two experiments using Llama.cpp and the QWEN model: running various cache configurations and evaluating their performance, and outputting a trace of the memory footprint. Using these experiments, we investigate the memory access patterns and performance characteristics to identify potential optimizations.