NuggetIndex: Governed Atomic Retrieval for Maintainable RAG
For practitioners building maintainable RAG systems, NuggetIndex addresses the unit mismatch between evaluation and retrieval, enabling temporal and conflict-aware retrieval in evolving corpora.
NuggetIndex introduces atomic, managed information units (nuggets) for RAG, improving nugget recall by 42%, temporal correctness by 9 percentage points, and reducing conflict rates by 55% while cutting generator input length by 64%.
Retrieval-augmented generation (RAG) systems are frequently evaluated via fact-based metrics, yet standard implementations retrieve passages or static propositions. This unit mismatch between evaluation and retrieval objects hinders maintenance when corpora evolve and fails to capture superseded facts or source disagreements. We propose NuggetIndex, a retrieval system that stores atomic information units as managed records, so called nuggets. Each record maintains links to evidence, a temporal validity interval, and a lifecycle state. By filtering invalid or deprecated nuggets prior to ranking, the system prevents the inclusion of outdated information. We evaluate the approach using a nuggetized MS MARCO subset, a temporal Wikipedia QA dataset, and a multi-hop QA task. Against passage and unmanaged proposition retrieval baselines, NuggetIndex improves nugget recall by 42%, increases temporal correctness by 9 percentage points without the recall collapse observed in time-filtered baselines, and reduces conflict rates by 55%. The compact nugget format reduces generator input length by 64% while enabling lightweight index structures suitable for browser-based and resource-constrained deployment. We release our implementation, datasets, and evaluation scripts