Towards Lighter and Robust Evaluation for Retrieval Augmented Generation
This addresses the need for cheaper and more transparent evaluation methods for RAG systems, which is incremental as it builds on existing evaluation frameworks.
The paper tackles the problem of evaluating hallucination in Retrieval Augmented Generation (RAG) systems by proposing a lightweight approach using smaller, quantized LLMs to provide accessible and interpretable metrics, resulting in a new AUC metric as an alternative to correlation with human judgment.
Large Language Models are prompting us to view more NLP tasks from a generative perspective. At the same time, they offer a new way of accessing information, mainly through the RAG framework. While there have been notable improvements for the autoregressive models, overcoming hallucination in the generated answers remains a continuous problem. A standard solution is to use commercial LLMs, such as GPT4, to evaluate these algorithms. However, such frameworks are expensive and not very transparent. Therefore, we propose a study which demonstrates the interest of open-weight models for evaluating RAG hallucination. We develop a lightweight approach using smaller, quantized LLMs to provide an accessible and interpretable metric that gives continuous scores for the generated answer with respect to their correctness and faithfulness. This score allows us to question decisions' reliability and explore thresholds to develop a new AUC metric as an alternative to correlation with human judgment.