Assessing generalization capability of text ranking models in Polish
This addresses the domain-specific problem of improving text ranking for Polish language applications, though it is incremental as it builds on existing reranking techniques.
The paper tackled the problem of reranking models for Polish language retrieval-augmented generation, finding that most models struggle with out-of-domain generalization, but their best model achieved a new state-of-the-art, outperforming existing models with up to 30 times more parameters.
Retrieval-augmented generation (RAG) is becoming an increasingly popular technique for integrating internal knowledge bases with large language models. In a typical RAG pipeline, three models are used, responsible for the retrieval, reranking, and generation stages. In this article, we focus on the reranking problem for the Polish language, examining the performance of rerankers and comparing their results with available retrieval models. We conduct a comprehensive evaluation of existing models and those trained by us, utilizing a benchmark of 41 diverse information retrieval tasks for the Polish language. The results of our experiments show that most models struggle with out-of-domain generalization. However, a combination of effective optimization method and a large training dataset allows for building rerankers that are both compact in size and capable of generalization. The best of our models establishes a new state-of-the-art for reranking in the Polish language, outperforming existing models with up to 30 times more parameters.