LumberChunker: Long-Form Narrative Document Segmentation
This addresses the need for better document segmentation in NLP retrieval tasks, particularly for narrative texts, though it is incremental as it builds on existing chunking methods.
The paper tackles the problem of segmenting long-form narrative documents for dense retrieval by proposing LumberChunker, a method that uses an LLM to dynamically identify content shifts, resulting in a 7.37% improvement in retrieval performance over the most competitive baseline on the introduced GutenQA benchmark.
Modern NLP tasks increasingly rely on dense retrieval methods to access up-to-date and relevant contextual information. We are motivated by the premise that retrieval benefits from segments that can vary in size such that a content's semantic independence is better captured. We propose LumberChunker, a method leveraging an LLM to dynamically segment documents, which iteratively prompts the LLM to identify the point within a group of sequential passages where the content begins to shift. To evaluate our method, we introduce GutenQA, a benchmark with 3000 "needle in a haystack" type of question-answer pairs derived from 100 public domain narrative books available on Project Gutenberg. Our experiments show that LumberChunker not only outperforms the most competitive baseline by 7.37% in retrieval performance (DCG@20) but also that, when integrated into a RAG pipeline, LumberChunker proves to be more effective than other chunking methods and competitive baselines, such as the Gemini 1.5M Pro. Our Code and Data are available at https://github.com/joaodsmarques/LumberChunker