Kelle: Co-design KV Caching and eDRAM for Efficient LLM Serving in Edge Computing
This work addresses the challenge of deploying LLMs on resource-constrained edge devices, which is crucial for low-latency and private AI applications, though it is incremental as it builds on existing KV caching and eDRAM techniques.
The paper tackles the problem of efficiently serving Large Language Models (LLMs) on edge devices by addressing the memory overhead of key-value (KV) caches, proposing a software-hardware co-design solution called Kelle that uses embedded DRAM (eDRAM) with optimized algorithms, resulting in a 3.9x speedup and 4.5x energy savings compared to baselines.
Running Large Language Models (LLMs) on edge devices is crucial for reducing latency, improving real-time processing, and enhancing privacy. By performing inference directly on the device, data does not need to be sent to the cloud, ensuring faster responses and reducing reliance on network connectivity. However, implementing LLMs on edge devices presents challenges, particularly with managing key-value (KV) caches, which plays a pivotal role in LLM serving. As the input text lengthens, the size of the KV cache increases linearly with the sequence length, leading to a significant memory footprint and data access costs. On the other hand, edge devices have limited memory and computational power, making it hard to store and efficiently access the large caches needed for LLM inference. To mitigate the substantial overhead caused by KV cache, we propose using embedded DRAM (eDRAM) as the primary storage for LLM serving in edge device, which offers higher storage density compared to SRAM. However, to ensure data integrity, eDRAM needs periodic refresh operations, which are power-intensive. To reduce eDRAM costs and improve overall system performance, we propose~\textit{Kelle}, a software-hardware co-design solution optimized for deploying LLMs on eDRAM-based edge systems. Combined with our fine-grained memory eviction, recomputation, and refresh control algorithms, the \textit{Kelle} accelerator delivers a $3.9\times$ speedup and $4.5\times$ energy savings compared to existing baseline solutions.