ICLERB: In-Context Learning Embedding and Reranker Benchmark
This addresses the need for specialized benchmarks and training strategies for retrieval in ICL, which is an incremental improvement for enhancing LLM accuracy in task-specific settings.
The paper tackled the problem of retrieval for In-Context Learning (ICL) by reframing it as a recommendation problem and introduced the ICLERB benchmark and RLRAIF algorithm, resulting in small models fine-tuned with RLRAIF outperforming large state-of-the-art retrieval models.
In-Context Learning (ICL) enables Large Language Models (LLMs) to perform new tasks by conditioning on prompts with relevant information. Retrieval-Augmented Generation (RAG) enhances ICL by incorporating retrieved documents into the LLM's context at query time. However, traditional retrieval methods focus on semantic relevance, treating retrieval as a search problem. In this paper, we propose reframing retrieval for ICL as a recommendation problem, aiming to select documents that maximize utility in ICL tasks. We introduce the In-Context Learning Embedding and Reranker Benchmark (ICLERB), a novel evaluation framework that compares retrievers based on their ability to enhance LLM accuracy in ICL settings. Additionally, we propose a novel Reinforcement Learning-to-Rank from AI Feedback (RLRAIF) algorithm, designed to fine-tune retrieval models using minimal feedback from the LLM. Our experimental results reveal notable differences between ICLERB and existing benchmarks, and demonstrate that small models fine-tuned with our RLRAIF algorithm outperform large state-of-the-art retrieval models. These findings highlight the limitations of existing evaluation methods and the need for specialized benchmarks and training strategies adapted to ICL.