An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation
This work addresses the challenge of complex and expensive RAG configuration optimization for practitioners, but it is incremental as it benchmarks existing methods rather than introducing new ones.
The study tackled the problem of optimizing Retrieval-Augmented Generation (RAG) configurations by benchmarking 5 hyper-parameter optimization algorithms across 5 datasets, showing that efficient methods like greedy or random search significantly boost performance for all datasets.
Finding the optimal Retrieval-Augmented Generation (RAG) configuration for a given use case can be complex and expensive. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To address this gap, we present a comprehensive study involving 5 HPO algorithms over 5 datasets from diverse domains, including a new one collected for this work on real-world product documentation. Our study explores the largest HPO search space considered to date, with three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the prevalent practice of optimizing according to RAG pipeline order.