How User Language Affects Conflict Fatality Estimates in ChatGPT
This reveals a novel bias mechanism in AI language models that could amplify media biases and reinforce conflicts, particularly for users in conflict zones.
The study investigated how ChatGPT's responses to queries about conflict fatalities vary by language, finding that GPT-3.5 provides 27±11% lower fatality estimates when queried in the attacker's language compared to the targeted group's language, with evasive answers exacerbating the bias.
OpenAI's ChatGPT language model has gained popularity as a powerful tool for complex problem-solving and information retrieval. However, concerns arise about the reproduction of biases present in the language-specific training data. In this study, we address this issue in the context of the Israeli-Palestinian and Turkish-Kurdish conflicts. Using GPT-3.5, we employed an automated query procedure to inquire about casualties in specific airstrikes, in both Hebrew and Arabic for the former conflict and Turkish and Kurdish for the latter. Our analysis reveals that GPT-3.5 provides 27$\pm$11 percent lower fatality estimates when queried in the language of the attacker than in the language of the targeted group. Evasive answers denying the existence of such attacks further increase the discrepancy, creating a novel bias mechanism not present in regular search engines. This language bias has the potential to amplify existing media biases and contribute to information bubbles, ultimately reinforcing conflicts.