DRAGON: Dynamic RAG Benchmark On News
This work addresses the problem of scarce and static evaluation resources for RAG systems in non-English languages like Russian, which is incremental as it extends existing benchmarking approaches to a new language and dynamic setting.
The authors tackled the lack of dynamic benchmarks for evaluating Retrieval-Augmented Generation (RAG) systems in Russian by creating DRAGON, a benchmark built on a regularly updated news corpus that supports comprehensive evaluation of retriever and generator components, including automatic question generation based on a Knowledge Graph.
Retrieval-Augmented Generation (RAG) is a widely adopted approach for improving the factuality of large language models (LLMs) by incorporating external knowledge at inference time. Although there exist multiple RAG benchmarks for English, evaluation resources for other languages, including Russian, remain scarce and static, failing to capture the dynamic nature of real-world deployments. In this work, we present DRAGON (Dynamic RAG Benchmark On News), the first dynamic benchmark for evaluating RAG systems in Russian on a changing news corpora. DRAGON is built upon a regularly updated corpus of Russian news and public documents and supports comprehensive evaluation of both the retriever and generator components. Question generation is performed automatically with the use of Knowledge Graph constructed from the corpus and enables the extraction of four core question types aligned with distinct subgraph patterns. We release a complete evaluation framework comprising the pipeline for automatic question generation, evaluation scripts, which are potentially reusable for other languages and multilingual settings, and benchmark data. We also launch a public leaderboard to encourage community participation and comparison.