Language Fairness in Multilingual Information Retrieval
This addresses fairness issues in MLIR for users and developers, but it is incremental as it builds on existing group fairness concepts by adapting them to a multilingual context without protected groups.
The paper tackles the problem of language bias in multilingual information retrieval (MLIR) systems, where documents in certain languages are unfairly ranked, and proposes a new fairness metric called PEER (Probability of Equal Expected Rank) to evaluate this, demonstrating its alignment with prior findings on real benchmarks.
Multilingual information retrieval (MLIR) considers the problem of ranking documents in several languages for a query expressed in a language that may differ from any of those languages. Recent work has observed that approaches such as combining ranked lists representing a single document language each or using multilingual pretrained language models demonstrate a preference for one language over others. This results in systematic unfair treatment of documents in different languages. This work proposes a language fairness metric to evaluate whether documents across different languages are fairly ranked through statistical equivalence testing using the Kruskal-Wallis test. In contrast to most prior work in group fairness, we do not consider any language to be an unprotected group. Thus our proposed measure, PEER (Probability of EqualExpected Rank), is the first fairness metric specifically designed to capture the language fairness of MLIR systems. We demonstrate the behavior of PEER on artificial ranked lists. We also evaluate real MLIR systems on two publicly available benchmarks and show that the PEER scores align with prior analytical findings on MLIR fairness. Our implementation is compatible with ir-measures and is available at http://github.com/hltcoe/peer_measure.