MPIC: Position-Independent Multimodal Context Caching System for Efficient MLLM Serving
This work addresses performance bottlenecks in MLLM serving for applications like multimodal retrieval-augmented generation, though it is incremental as it builds on existing caching techniques.
The paper tackles the inefficiency of existing context caching systems for Multimodal Large Language Models (MLLMs), which recompute full KV caches for slightly differing prefixes, and proposes MPIC, a position-independent caching system that reduces response time by up to 54% and improves throughput by 2× while maintaining negligible accuracy loss.
The context caching technique is employed to accelerate the Multimodal Large Language Model (MLLM) inference by prevailing serving platforms currently. However, this approach merely reuses the Key-Value (KV) cache of the initial sequence of prompt, resulting in full KV cache recomputation even if the prefix differs slightly. This becomes particularly inefficient in the context of interleaved text and images, as well as multimodal retrieval-augmented generation. This paper proposes position-independent caching as a more effective approach for multimodal information management. We have designed and implemented a caching system, named MPIC, to address both system-level and algorithm-level challenges. MPIC stores the KV cache on local disks when receiving multimodal data, and calculates and loads the KV cache in parallel during inference. To mitigate accuracy degradation, we have incorporated the integrated reuse and recompute mechanism within the system. The experimental results demonstrate that MPIC can achieve up to 54\% reduction in response time and 2$\times$ improvement in throughput compared to existing context caching systems, while maintaining negligible or no accuracy loss.