Big City Bias: Evaluating the Impact of Metropolitan Size on Computational Job Market Abilities of Language Models
This addresses bias in LLMs for job matching, which could impact fairness in employment opportunities, but is incremental as it focuses on quantifying an existing known bias.
The study quantified metropolitan size bias in large language models (LLMs) for job matching tasks, finding that smaller regions are underrepresented with up to 300% worse performance in benchmarks like salary and commute predictions compared to larger regions.
Large language models (LLMs) have emerged as a useful technology for job matching, for both candidates and employers. Job matching is often based on a particular geographic location, such as a city or region. However, LLMs have known biases, commonly derived from their training data. In this work, we aim to quantify the metropolitan size bias encoded within large language models, evaluating zero-shot salary, employer presence, and commute duration predictions in 384 of the United States' metropolitan regions. Across all benchmarks, we observe negative correlations between the metropolitan size and the performance of the LLMS, indicating that smaller regions are indeed underrepresented. More concretely, the smallest 10 metropolitan regions show upwards of 300% worse benchmark performance than the largest 10.