mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Dataset
This work addresses the problem of limited multilingual resources for information retrieval researchers, though it is incremental as it builds on existing datasets and translation methods.
The authors tackled the scarcity of multilingual ranking datasets by creating mMARCO, a machine-translated version of MS MARCO in 13 languages, and showed that models finetuned on it achieve superior zero-shot effectiveness on Mr. TyDi compared to English-only training, with a distilled reranker being competitive while having 5.4 times fewer parameters.
The MS MARCO ranking dataset has been widely used for training deep learning models for IR tasks, achieving considerable effectiveness on diverse zero-shot scenarios. However, this type of resource is scarce in languages other than English. In this work, we present mMARCO, a multilingual version of the MS MARCO passage ranking dataset comprising 13 languages that was created using machine translation. We evaluated mMARCO by finetuning monolingual and multilingual reranking models, as well as a multilingual dense retrieval model on this dataset. We also evaluated models finetuned using the mMARCO dataset in a zero-shot scenario on Mr. TyDi dataset, demonstrating that multilingual models finetuned on our translated dataset achieve superior effectiveness to models finetuned on the original English version alone. Our experiments also show that a distilled multilingual reranker is competitive with non-distilled models while having 5.4 times fewer parameters. Lastly, we show a positive correlation between translation quality and retrieval effectiveness, providing evidence that improvements in translation methods might lead to improvements in multilingual information retrieval. The translated datasets and finetuned models are available at https://github.com/unicamp-dl/mMARCO.