Evaluating Chunking Strategies for Retrieval-Augmented Generation on Academic Texts
This is an incremental evaluation of chunking strategies for RAG on academic texts, providing limited practical guidance due to the negative result and reliability issues with the evaluation metric.
The paper evaluates cluster-based semantic chunking against fixed-size and recursive chunking for RAG on academic theses, finding that cluster-based chunking does not outperform simpler strategies under the tested configuration.
Retrieval-Augmented Generation (RAG) systems use the question-answering capabilities of Large Language Models (LLMs) to access information outside their parameters. We evaluate if cluster-based semantic chunking improves retrieval and answer quality compared to fixed-size and recursive chunking evaluating on long, structured academic theses using the Retrieval Augmented Generation Assessment (RAGAs) framework. RAGAs based faithfulness shows limited reliability in this setup. Performance on fixed versus document specific questions varied substantially, likely related to the formatting of documents and preprocessing. Under the tested configuration, cluster-based chunking did not outperform simpler strategies.