A New HOPE: Domain-agnostic Automatic Evaluation of Text Chunking
This addresses the problem of optimizing document chunking for RAG systems, which is crucial for improving factual accuracy in AI responses, though it is incremental as it builds on existing RAG methods.
The paper tackles the lack of a framework to analyze how different text chunking methods impact Retrieval-Augmented Generation (RAG) performance by introducing HOPE, a domain-agnostic automatic evaluation metric that quantifies chunking characteristics, showing it correlates with RAG performance and revealing semantic independence between passages can improve factual correctness by up to 56.2%.
Document chunking fundamentally impacts Retrieval-Augmented Generation (RAG) by determining how source materials are segmented before indexing. Despite evidence that Large Language Models (LLMs) are sensitive to the layout and structure of retrieved data, there is currently no framework to analyze the impact of different chunking methods. In this paper, we introduce a novel methodology that defines essential characteristics of the chunking process at three levels: intrinsic passage properties, extrinsic passage properties, and passages-document coherence. We propose HOPE (Holistic Passage Evaluation), a domain-agnostic, automatic evaluation metric that quantifies and aggregates these characteristics. Our empirical evaluations across seven domains demonstrate that the HOPE metric correlates significantly (p > 0.13) with various RAG performance indicators, revealing contrasts between the importance of extrinsic and intrinsic properties of passages. Semantic independence between passages proves essential for system performance with a performance gain of up to 56.2% in factual correctness and 21.1% in answer correctness. On the contrary, traditional assumptions about maintaining concept unity within passages show minimal impact. These findings provide actionable insights for optimizing chunking strategies, thus improving RAG system design to produce more factually correct responses.