How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG
This work addresses evaluation flaws in GraphRAG research, which is crucial for ensuring reliable progress in retrieval-augmented generation for AI applications.
The authors tackled the problem of biased performance evaluation in GraphRAG methods by proposing an unbiased evaluation framework, which revealed that the performance gains of three representative methods are much more moderate than previously reported.
By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions. Many GraphRAG methods have been proposed and reported inspiring performance in answer quality. However, we observe that the current answer evaluation framework for GraphRAG has two critical flaws, i.e., unrelated questions and evaluation biases, which may lead to biased or even wrong conclusions on performance. To tackle the two flaws, we propose an unbiased evaluation framework that uses graph-text-grounded question generation to produce questions that are more related to the underlying dataset and an unbiased evaluation procedure to eliminate the biases in LLM-based answer assessment. We apply our unbiased framework to evaluate 3 representative GraphRAG methods and find that their performance gains are much more moderate than reported previously. Although our evaluation framework may still have flaws, it calls for scientific evaluations to lay solid foundations for GraphRAG research.