ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval
This work addresses the challenge of ranking relevant documents at the top for retrieval-augmented generation systems, which is crucial for enhancing question answering performance, though it is incremental as it builds on existing re-ranking and large language model techniques.
The paper tackles the problem of improving document retrieval for open-domain question answering by introducing ASRank, a zero-shot re-ranking method that uses answer scent to score documents, resulting in significant accuracy improvements such as increasing Top-1 retrieval accuracy on NQ from 19.2% to 46.5% for MSS.
Retrieval-Augmented Generation (RAG) models have drawn considerable attention in modern open-domain question answering. The effectiveness of RAG depends on the quality of the top retrieved documents. However, conventional retrieval methods sometimes fail to rank the most relevant documents at the top. In this paper, we introduce ASRank, a new re-ranking method based on scoring retrieved documents using zero-shot answer scent which relies on a pre-trained large language model to compute the likelihood of the document-derived answers aligning with the answer scent. Our approach demonstrates marked improvements across several datasets, including NQ, TriviaQA, WebQA, ArchivalQA, HotpotQA, and Entity Questions. Notably, ASRank increases Top-1 retrieval accuracy on NQ from $19.2\%$ to $46.5\%$ for MSS and $22.1\%$ to $47.3\%$ for BM25. It also shows strong retrieval performance on several datasets compared to state-of-the-art methods (47.3 Top-1 by ASRank vs 35.4 by UPR by BM25).