Don't Overlook the Grammatical Gender: Bias Evaluation for Hindi-English Machine Translation
This addresses bias evaluation for non-English source languages in machine translation, which is incremental as it extends existing methods to Hindi with a focus on grammatical gender.
The paper tackled gender bias in Hindi-English neural machine translation by creating gender-specific sentence sets to evaluate if models can discern gender from grammatical cues, finding that existing systems often fail to accurately translate gender when relying on biased associations rather than grammatical markers.
Neural Machine Translation (NMT) models, though state-of-the-art for translation, often reflect social biases, particularly gender bias. Existing evaluation benchmarks primarily focus on English as the source language of translation. For source languages other than English, studies often employ gender-neutral sentences for bias evaluation, whereas real-world sentences frequently contain gender information in different forms. Therefore, it makes more sense to evaluate for bias using such source sentences to determine if NMT models can discern gender from the grammatical gender cues rather than relying on biased associations. To illustrate this, we create two gender-specific sentence sets in Hindi to automatically evaluate gender bias in various Hindi-English (HI-EN) NMT systems. We emphasise the significance of tailoring bias evaluation test sets to account for grammatical gender markers in the source language.