ConGA: Guidelines for Contextual Gender Annotation. A Framework for Annotating Gender in Machine Translation
This work addresses gender bias in machine translation, providing a methodology and benchmark for more gender-aware NLP systems, though it is incremental as it builds on existing datasets and annotation practices.
The paper tackled the challenge of gender bias in machine translation from gender-neutral to gendered languages, such as English to Italian, by developing the ConGA framework for contextual gender annotation, which revealed systematic masculine overuse and inconsistent feminine realization in current systems.
Handling gender across languages remains a persistent challenge for Machine Translation (MT) and Large Language Models (LLMs), especially when translating from gender-neutral languages into morphologically gendered ones, such as English to Italian. English largely omits grammatical gender, while Italian requires explicit agreement across multiple grammatical categories. This asymmetry often leads MT systems to default to masculine forms, reinforcing bias and reducing translation accuracy. To address this issue, we present the Contextual Gender Annotation (ConGA) framework, a linguistically grounded set of guidelines for word-level gender annotation. The scheme distinguishes between semantic gender in English through three tags, Masculine (M), Feminine (F), and Ambiguous (A), and grammatical gender realisation in Italian (Masculine (M), Feminine (F)), combined with entity-level identifiers for cross-sentence tracking. We apply ConGA to the gENder-IT dataset, creating a gold-standard resource for evaluating gender bias in translation. Our results reveal systematic masculine overuse and inconsistent feminine realisation, highlighting persistent limitations of current MT systems. By combining fine-grained linguistic annotation with quantitative evaluation, this work offers both a methodology and a benchmark for building more gender-aware and multilingual NLP systems.