Demographic Biases and Gaps in the Perception of Sexism in Large Language Models
This work addresses the challenge of demographic biases in automated sexism detection for social media analysis, but it is incremental as it builds on prior efforts to improve detection without introducing a new method.
The study tackled the problem of detecting sexism in social media text using Large Language Models (LLMs), finding that while LLMs can detect sexism to some extent based on overall population opinions, they fail to accurately replicate the diverse perceptions across different demographic groups, such as age and gender, using the EXIST 2024 tweet dataset.
The use of Large Language Models (LLMs) has proven to be a tool that could help in the automatic detection of sexism. Previous studies have shown that these models contain biases that do not accurately reflect reality, especially for minority groups. Despite various efforts to improve the detection of sexist content, this task remains a significant challenge due to its subjective nature and the biases present in automated models. We explore the capabilities of different LLMs to detect sexism in social media text using the EXIST 2024 tweet dataset. It includes annotations from six distinct profiles for each tweet, allowing us to evaluate to what extent LLMs can mimic these groups' perceptions in sexism detection. Additionally, we analyze the demographic biases present in the models and conduct a statistical analysis to identify which demographic characteristics (age, gender) contribute most effectively to this task. Our results show that, while LLMs can to some extent detect sexism when considering the overall opinion of populations, they do not accurately replicate the diversity of perceptions among different demographic groups. This highlights the need for better-calibrated models that account for the diversity of perspectives across different populations.