LoRE: Logit-Ranked Retriever Ensemble for Enhancing Open-Domain Question Answering
This addresses suboptimal answer generation in open-domain question answering systems, representing a strong specific gain.
The paper tackled the problem of positional bias in retrieval-based question answering systems by proposing LoRE, which improved answer accuracy and relevance, achieving up to 22.83% improvement in exact match scores on SQuAD.
Retrieval-based question answering systems often suffer from positional bias, leading to suboptimal answer generation. We propose LoRE (Logit-Ranked Retriever Ensemble), a novel approach that improves answer accuracy and relevance by mitigating positional bias. LoRE employs an ensemble of diverse retrievers, such as BM25 and sentence transformers with FAISS indexing. A key innovation is a logit-based answer ranking algorithm that combines the logit scores from a large language model (LLM), with the retrieval ranks of the passages. Experimental results on NarrativeQA, SQuAD demonstrate that LoRE significantly outperforms existing retrieval-based methods in terms of exact match and F1 scores. On SQuAD, LoRE achieves 14.5\%, 22.83\%, and 14.95\% improvements over the baselines for ROUGE-L, EM, and F1, respectively. Qualitatively, LoRE generates more relevant and accurate answers, especially for complex queries.