Retro-li: Small-Scale Retrieval Augmented Generation Supporting Noisy Similarity Searches and Domain Shift Generalization
This work addresses the problem of deploying efficient RAG systems in resource-constrained or noisy environments for AI practitioners, though it represents an incremental improvement over existing RAG methods.
The authors tackled the challenge of making retrieval-augmented generation (RAG) systems effective with small-scale databases, which require more accurate similarity searches due to sparser memory. They introduced Retro-li, which reduces perplexity by up to 15% with noisy searches and improves domain shift generalization while maintaining minimal (<1%) performance loss when implemented on analog hardware.
The retrieval augmented generation (RAG) system such as Retro has been shown to improve language modeling capabilities and reduce toxicity and hallucinations by retrieving from a database of non-parametric memory containing trillions of entries. We introduce Retro-li that shows retrieval can also help using a small-scale database, but it demands more accurate and better neighbors when searching in a smaller hence sparser non-parametric memory. This can be met by using a proper semantic similarity search. We further propose adding a regularization to the non-parametric memory for the first time: it significantly reduces perplexity when the neighbor search operations are noisy during inference, and it improves generalization when a domain shift occurs. We also show that Retro-li's non-parametric memory can potentially be implemented on analog in-memory computing hardware, exhibiting O(1) search time while causing noise in retrieving neighbors, with minimal (<1%) performance loss. Our code is available at: https://github.com/IBM/Retrieval-Enhanced-Transformer-Little.