VectorLiteRAG: Latency-Aware and Fine-Grained Resource Partitioning for Efficient RAG
This addresses efficiency challenges for deploying RAG systems in resource-constrained environments, though it is incremental as it builds on existing RAG frameworks.
The paper tackles the performance degradation in RAG systems from co-locating vector search and LLMs on shared GPUs by introducing VectorLiteRAG, which uses fine-grained resource partitioning to optimize latency and throughput, achieving up to 1.5 times higher SLO-compliant throughput without extra hardware.
Retrieval-Augmented Generation (RAG) systems combine vector similarity search with large language models (LLMs) to deliver accurate, context-aware responses. However, co-locating the vector retriever and the LLM on shared GPU infrastructure introduces significant challenges: vector search is memory and I/O intensive, while LLM inference demands high throughput and low latency. Naive resource sharing often leads to severe performance degradation, particularly under high request load or large index sizes. We present VectorLiteRAG, a deployment-friendly RAG system that achieves latency-compliant inference without requiring additional hardware resources. VectorLiteRAG introduces a fine-grained GPU resource allocation mechanism based on detailed performance modeling and access pattern analysis. By estimating search latency and query hit rate distributions, it identifies an optimal index partitioning point across CPU and GPU tiers to minimize contention and maximize throughput. Our evaluations show that VectorLiteRAG consistently expands the SLO compliant request rate range across all tested configurations, including both small and large LLMs, and small and large vector databases compared to naive baselines and state of the art alternatives. In the best case, VectorLiteRAG improves the attainable SLO throughput by up to 1.5 times without compromising generation quality or requiring additional compute resources.