Graph-Aware Late Chunking for Retrieval-Augmented Generation in Biomedical Literature
This work addresses the incomplete evaluation of RAG systems for biomedical literature, highlighting a key open problem in multi-section synthesis, though it is incremental as it builds on existing RAG frameworks.
The paper tackled the problem that standard ranking metrics like MRR inadequately evaluate retrieval-augmented generation (RAG) systems for biomedical literature by focusing on precision over breadth, and found that structure-aware methods retrieve from up to 15.6x more sections than content-similarity methods, with KG-infused retrieval narrowing the answer-quality gap to delta-F1 = 0.009 while maintaining 4.6x section diversity.
Retrieval-Augmented Generation (RAG) systems for biomedical literature are typically evaluated using ranking metrics like Mean Reciprocal Rank (MRR), which measure how well the system identifies the single most relevant chunk. We argue that for full-text scientific documents, this paradigm is incomplete: it rewards retrieval precision while ignoring retrieval breadth -- the ability to surface evidence from across a document's structural sections. We propose GraLC-RAG, a framework that unifies late chunking with graph-aware structural intelligence, introducing structure-aware chunk boundary detection, UMLS knowledge graph infusion, and graph-guided hybrid retrieval. We evaluate six strategies on 2,359 IMRaD-filtered PubMed Central articles using 2,033 cross-section questions and two metric families: standard ranking metrics (MRR, Recall@k) and structural coverage metrics (SecCov@k, CS Recall). Our results expose a sharp divergence: content-similarity methods achieve the highest MRR (0.517) but always retrieve from a single section, while structure-aware methods retrieve from up to 15.6x more sections. Generation experiments show that KG-infused retrieval narrows the answer-quality gap to delta-F1 = 0.009 while maintaining 4.6x section diversity. These findings demonstrate that standard metrics systematically undervalue structural retrieval and that closing the multi-section synthesis gap is a key open problem for biomedical RAG.