Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines
This addresses the challenge of zero-shot performance for AI models in specific domains, though it is incremental as it builds on existing retrieval-augmented methods.
The paper tackles the problem of pre-trained models lacking domain-specific nuances by augmenting them with search engine retrieval at test time based on uncertain cases, resulting in a 15 percentage point accuracy increase on datasets like Stanford Cars and Flowers.
Large pre-trained models can dramatically reduce the amount of task-specific data required to solve a problem, but they often fail to capture domain-specific nuances out of the box. The Web likely contains the information necessary to excel on any specific application, but identifying the right data a priori is challenging. This paper shows how to leverage recent advances in NLP and multi-modal learning to augment a pre-trained model with search engine retrieval. We propose to retrieve useful data from the Web at test time based on test cases that the model is uncertain about. Different from existing retrieval-augmented approaches, we then update the model to address this underlying uncertainty. We demonstrate substantial improvements in zero-shot performance, e.g. a remarkable increase of 15 percentage points in accuracy on the Stanford Cars and Flowers datasets. We also present extensive experiments that explore the impact of noisy retrieval and different learning strategies.