Evaluating Chunking Strategies For Retrieval-Augmented Generation in Oil and Gas Enterprise Documents
This addresses document retrieval quality for oil and gas enterprises using RAG, though it's incremental as it compares existing chunking methods on new domain data.
The paper evaluated four document chunking strategies for Retrieval-Augmented Generation (RAG) using oil and gas enterprise documents, finding that structure-aware chunking achieved higher retrieval effectiveness with lower computational costs, but all methods performed poorly on visual/spatial documents like piping and instrumentation diagrams.
Retrieval-Augmented Generation (RAG) has emerged as a framework to address the constraints of Large Language Models (LLMs). Yet, its effectiveness fundamentally hinges on document chunking - an often-overlooked determinant of its quality. This paper presents an empirical study quantifying performance differences across four chunking strategies: fixed-size sliding window, recursive, breakpoint-based semantic, and structure-aware. We evaluated these methods using a proprietary corpus of oil and gas enterprise documents, including text-heavy manuals, table-heavy specifications, and piping and instrumentation diagrams (P and IDs). Our findings show that structure-aware chunking yields higher overall retrieval effectiveness, particularly in top-K metrics, and incurs significantly lower computational costs than semantic or baseline strategies. Crucially, all four methods demonstrated limited effectiveness on P and IDs, underscoring a core limitation of purely text-based RAG within visually and spatially encoded documents. We conclude that while explicit structure preservation is essential for specialised domains, future work must integrate multimodal models to overcome current limitations.