RAGe: A Retrieval-Augmented Generation Evaluation Framework
For developers deploying RAG systems, it reduces manual tuning by correlating accuracy, efficiency, and scalability with hardware constraints.
RAGe is a modular framework for benchmarking RAG applications that recommends optimal pipeline components (chunking, embeddings, retrievers) based on resource telemetry, enabling efficient deployment on consumer-grade hardware.
Deploying Large Language Model (LLM) applications, particularly those relying on Retrieval-Augmented Generation (RAG), remains challenging due to high computational demands, outdated knowledge bases, and the need to manually select optimal pipeline components. In this work, we propose a modular framework for benchmarking and guiding the efficient development of RAG applications by focusing on resource telemetry and component recommendation, suggesting the best components for a domain-specific dataset. Our approach leverages core techniques in LLM applications, including document chunking, vector databases, embedding models, and retrievers, to evaluate trade-offs among accuracy, efficiency, and scalability. By directly correlating retrieval and generation quality with underlying hardware constraints, RAGe supports researchers to identify the most effective, domain-specific RAG setups for their specific operational needs, facilitating rapid prototyping even on consumer-grade hardware.