Investigating Task Arithmetic for Zero-Shot Information Retrieval
This work addresses the challenge of adapting retrieval models to new tasks and domains without additional training, which is significant for researchers and practitioners in NLP and information retrieval, though it is incremental as it builds on existing Task Arithmetic techniques.
The paper tackles the problem of large language models degrading in zero-shot performance on unseen tasks and domains for information retrieval by investigating Task Arithmetic, which combines pre-trained model weights via mathematical operations to adapt retrieval models without fine-tuning, resulting in improvements of up to 18% in NDCG@10 and 15% in P@10 on scientific, biomedical, and multilingual datasets.
Large Language Models (LLMs) have shown impressive zero-shot performance across a variety of Natural Language Processing tasks, including document re-ranking. However, their effectiveness degrades on unseen tasks and domains, largely due to shifts in vocabulary and word distributions. In this paper, we investigate Task Arithmetic, a technique that combines the weights of LLMs pre-trained on different tasks or domains via simple mathematical operations, such as addition or subtraction, to adapt retrieval models without requiring additional fine-tuning. Our method is able to synthesize diverse tasks and domain knowledge into a single model, enabling effective zero-shot adaptation in different retrieval contexts. Extensive experiments on publicly available scientific, biomedical, and multilingual datasets show that our method improves state-of-the-art re-ranking performance by up to 18% in NDCG@10 and 15% in P@10. In addition to these empirical gains, our analysis provides insights into the strengths and limitations of Task Arithmetic as a practical strategy for zero-shot learning and model adaptation. We make our code publicly available at https://github.com/DetectiveMB/Task-Arithmetic-for-ZS-IR.