Towards Best Practices for Training Multilingual Dense Retrieval Models
This work offers practical guidance for practitioners building search applications, especially for low-resource languages, but it is incremental as it synthesizes existing methods into best practices without introducing a new paradigm.
The paper tackles the problem of training multilingual dense retrieval models by providing best practices for three scenarios: when multilingual transformers are available without relevance judgments, when both models and training data are available, and when training data is available but models are not, resulting in insights into multi-stage fine-tuning, cross-lingual transfer, and the use of out-of-language data.
Dense retrieval models using a transformer-based bi-encoder design have emerged as an active area of research. In this work, we focus on the task of monolingual retrieval in a variety of typologically diverse languages using one such design. Although recent work with multilingual transformers demonstrates that they exhibit strong cross-lingual generalization capabilities, there remain many open research questions, which we tackle here. Our study is organized as a "best practices" guide for training multilingual dense retrieval models, broken down into three main scenarios: where a multilingual transformer is available, but relevance judgments are not available in the language of interest; where both models and training data are available; and, where training data are available not but models. In considering these scenarios, we gain a better understanding of the role of multi-stage fine-tuning, the strength of cross-lingual transfer under various conditions, the usefulness of out-of-language data, and the advantages of multilingual vs. monolingual transformers. Our recommendations offer a guide for practitioners building search applications, particularly for low-resource languages, and while our work leaves open a number of research questions, we provide a solid foundation for future work.