A Systematic Analysis of Chunking Strategies for Reliable Question Answering
This provides actionable insights for cost-efficient deployment of RAG systems in industry, though it is incremental as it systematically evaluates existing methods rather than introducing new ones.
The study tackled how document chunking strategies affect the reliability of Retrieval-Augmented Generation systems in industry, finding that sentence chunking is the most cost-effective method, matching semantic chunking up to ~5k tokens, and identifying a 'context cliff' that reduces quality beyond ~2.5k tokens.
We study how document chunking choices impact the reliability of Retrieval-Augmented Generation (RAG) systems in industry. While practice often relies on heuristics, our end-to-end evaluation on Natural Questions systematically varies chunking method (token, sentence, semantic, code), chunk size, overlap, and context length. We use a standard industrial setup: SPLADE retrieval and a Mistral-8B generator. We derive actionable lessons for cost-efficient deployment: (i) overlap provides no measurable benefit and increases indexing cost; (ii) sentence chunking is the most cost-effective method, matching semantic chunking up to ~5k tokens; (iii) a "context cliff" reduces quality beyond ~2.5k tokens; and (iv) optimal context depends on the goal (semantic quality peaks at small contexts; exact match at larger ones).