ExaRanker-Open: Synthetic Explanation for IR using Open-Source LLMs
This addresses data scarcity in IR for researchers and practitioners, though it is incremental as it extends prior work with open-source models.
The paper tackles the challenge of limited labeled examples in information retrieval by adapting the ExaRanker approach to use open-source LLMs for generating synthetic explanations, finding that this data augmentation consistently enhances neural rankers with benefits increasing with LLM size, achieving a 0.6 nDCG@10 improvement over baselines.
ExaRanker recently introduced an approach to training information retrieval (IR) models, incorporating natural language explanations as additional labels. The method addresses the challenge of limited labeled examples, leading to improvements in the effectiveness of IR models. However, the initial results were based on proprietary language models such as GPT-3.5, which posed constraints on dataset size due to its cost and data privacy. In this paper, we introduce ExaRanker-Open, where we adapt and explore the use of open-source language models to generate explanations. The method has been tested using different LLMs and datasets sizes to better comprehend the effective contribution of data augmentation. Our findings reveal that incorporating explanations consistently enhances neural rankers, with benefits escalating as the LLM size increases. Notably, the data augmentation method proves advantageous even with large datasets, as evidenced by ExaRanker surpassing the target baseline by 0.6 nDCG@10 points in our study. To encourage further advancements by the research community, we have open-sourced both the code and datasets at https://github.com/unicamp-dl/ExaRanker.