ProMoE: Fast MoE-based LLM Serving using Proactive Caching
This work addresses the problem of slow LLM serving on memory-constrained edge devices, offering a significant performance improvement for real-time applications.
The paper tackles the performance overhead of cache-based offloading in Mixture-of-Experts (MoE) models for large language models on edge devices by introducing ProMoE, a proactive caching system that predicts expert usage to eliminate cache misses, resulting in average speedups of 2.20x in prefill and 2.07x in decode stages compared to existing solutions.
The promising applications of large language models are often limited by the constrained GPU memory capacity available on edge devices. Mixture-of-Experts (MoE) models help address this issue by activating only a subset of the model's parameters during computation. This approach allows the unused parameters to be offloaded to host memory, thereby reducing the overall GPU memory demand. However, existing cache-based offloading solutions handle cache misses reactively, which significantly impacts system performance. In this paper, we introduce ProMoE, a novel proactive caching system that utilizes intermediate results to predict subsequent expert usage. By proactively fetching experts in advance, ProMoE eliminates passive cache misses, removes loading time from the critical path, and reduces the performance overhead associated with offloading. Our evaluations demonstrate that ProMoE achieves an average speedup of 2.20x (up to 3.21x) and 2.07x (up to 5.02x) in the prefill and decode stages, respectively, compared to existing offloading solutions.