Measuring Gender Bias in Job Title Matching for Grammatical Gender Languages
This addresses fairness issues in automated job ranking for users in grammatical gender languages, though it is incremental as it builds on existing bias evaluation methods.
The study tackled gender bias in job title matching systems for grammatical gender languages by proposing metrics and test sets to evaluate bias, finding that all tested multilingual models exhibited varying degrees of gender bias.
This work sets the ground for studying how explicit grammatical gender assignment in job titles can affect the results of automatic job ranking systems. We propose the usage of metrics for ranking comparison controlling for gender to evaluate gender bias in job title ranking systems, in particular RBO (Rank-Biased Overlap). We generate and share test sets for a job title matching task in four grammatical gender languages, including occupations in masculine and feminine form and annotated by gender and matching relevance. We use the new test sets and the proposed methodology to evaluate the gender bias of several out-of-the-box multilingual models to set as baselines, showing that all of them exhibit varying degrees of gender bias.