Augmenting Zero-Shot Dense Retrievers with Plug-in Mixture-of-Memories
This addresses the problem of robust zero-shot retrieval for AI systems, offering a novel approach to enhance generalization without retraining, though it is incremental in building on existing dense retrieval frameworks.
The paper tackles improving zero-shot generalization in dense retrieval by introducing Mixture-Of-Memory Augmentation (MoMA), which retrieves documents from multiple corpora and allows plugging in new memory at inference, resulting in strong zero-shot retrieval accuracy on the BEIR benchmark and outperforming methods relying on increased model parameters.
In this paper we improve the zero-shot generalization ability of language models via Mixture-Of-Memory Augmentation (MoMA), a mechanism that retrieves augmentation documents from multiple information corpora ("external memories"), with the option to "plug in" new memory at inference time. We develop a joint learning mechanism that trains the augmentation component with latent labels derived from the end retrieval task, paired with hard negatives from the memory mixture. We instantiate the model in a zero-shot dense retrieval setting by augmenting a strong T5-based retriever with MoMA. Our model, MoMA, obtains strong zero-shot retrieval accuracy on the eighteen tasks included in the standard BEIR benchmark. It outperforms systems that seek generalization from increased model parameters and computation steps. Our analysis further illustrates the necessity of augmenting with mixture-of-memory for robust generalization, the benefits of augmentation learning, and how MoMA utilizes the plug-in memory at inference time without changing its parameters. We plan to open source our code.