"The Dentist is an involved parent, the bartender is not": Revealing Implicit Biases in QA with Implicit BBQ
This addresses a critical oversight in fairness evaluation for NLP by providing a tool to detect implicit biases, which is important for researchers and practitioners working on AI ethics and bias mitigation.
The authors tackled the problem of evaluating implicit biases in large language models (LLMs) by introducing ImplicitBBQ, a benchmark that extends the Bias Benchmark for QA with implicitly cued protected attributes. Their evaluation of GPT-4o revealed a performance decline of up to 7% in accuracy for the 'sexual orientation' subcategory, indicating that current LLMs contain biases undetected by explicit benchmarks.
Existing benchmarks evaluating biases in large language models (LLMs) primarily rely on explicit cues, declaring protected attributes like religion, race, gender by name. However, real-world interactions often contain implicit biases, inferred subtly through names, cultural cues, or traits. This critical oversight creates a significant blind spot in fairness evaluation. We introduce ImplicitBBQ, a benchmark extending the Bias Benchmark for QA (BBQ) with implicitly cued protected attributes across 6 categories. Our evaluation of GPT-4o on ImplicitBBQ illustrates troubling performance disparity from explicit BBQ prompts, with accuracy declining up to 7% in the "sexual orientation" subcategory and consistent decline located across most other categories. This indicates that current LLMs contain implicit biases undetected by explicit benchmarks. ImplicitBBQ offers a crucial tool for nuanced fairness evaluation in NLP.