FLEX: A Benchmark for Evaluating Robustness of Fairness in Large Language Models
This addresses the need for more rigorous safety evaluations in LLMs to prevent harmful societal impacts, though it is incremental as it builds on existing fairness benchmarking.
The paper tackles the problem of social biases in Large Language Models (LLMs) by introducing FLEX, a benchmark designed to test fairness robustness under adversarial prompts, showing that traditional evaluations underestimate these risks.
Recent advancements in Large Language Models (LLMs) have significantly enhanced interactions between users and models. These advancements concurrently underscore the need for rigorous safety evaluations due to the manifestation of social biases, which can lead to harmful societal impacts. Despite these concerns, existing benchmarks may overlook the intrinsic weaknesses of LLMs, which can generate biased responses even with simple adversarial instructions. To address this critical gap, we introduce a new benchmark, Fairness Benchmark in LLM under Extreme Scenarios (FLEX), designed to test whether LLMs can sustain fairness even when exposed to prompts constructed to induce bias. To thoroughly evaluate the robustness of LLMs, we integrate prompts that amplify potential biases into the fairness assessment. Comparative experiments between FLEX and existing benchmarks demonstrate that traditional evaluations may underestimate the inherent risks in models. This highlights the need for more stringent LLM evaluation benchmarks to guarantee safety and fairness.