Whose wife is it anyway? Assessing bias against same-gender relationships in machine translation
This addresses bias in NLP systems for users affected by social relationship errors, but it is incremental as it extends existing gender bias research to a specific domain.
The paper tackled the problem of bias against same-gender relationships in machine translation systems, finding that three popular services consistently fail to accurately translate such sentences, with error rates varying by context and lower accuracy for high female-representation occupations.
Machine translation often suffers from biased data and algorithms that can lead to unacceptable errors in system output. While bias in gender norms has been investigated, less is known about whether MT systems encode bias about social relationships, e.g., "the lawyer kissed her wife." We investigate the degree of bias against same-gender relationships in MT systems, using generated template sentences drawn from several noun-gender languages (e.g., Spanish) and comprised of popular occupation nouns. We find that three popular MT services consistently fail to accurately translate sentences concerning relationships between entities of the same gender. The error rate varies considerably based on the context, and same-gender sentences referencing high female-representation occupations are translated with lower accuracy. We provide this work as a case study in the evaluation of intrinsic bias in NLP systems with respect to social relationships.