Improving the Serving Performance of Multi-LoRA Large Language Models via Efficient LoRA and KV Cache Management
This work improves inference efficiency for task-specific LLM applications, representing an incremental advancement in Multi-LoRA serving systems.
The paper tackles the problem of optimizing serving performance for Multi-LoRA LLMs by addressing inefficiencies in caching LoRA adapters and KV caches, resulting in a 63.4% average reduction in Time-To-First-Token compared to state-of-the-art methods.
Multiple Low-Rank Adapters (Multi-LoRAs) are gaining popularity for task-specific Large Language Model (LLM) applications. For multi-LoRA serving, caching hot KV caches and LoRA adapters in high bandwidth memory of accelerations can improve inference performance. However, existing Multi-LoRA inference systems fail to optimize serving performance like Time-To-First-Toke (TTFT), neglecting usage dependencies when caching LoRAs and KVs. We therefore propose FASTLIBRA, a Multi-LoRA caching system to optimize the serving performance. FASTLIBRA comprises a dependency-aware cache manager and a performance-driven cache swapper. The cache manager maintains the usage dependencies between LoRAs and KV caches during the inference with a unified caching pool. The cache swapper determines the swap-in or out of LoRAs and KV caches based on a unified cost model, when the HBM is idle or busy, respectively. Experimental results show that ELORA reduces the TTFT by 63.4% on average, compared to state-of-the-art works.