How Safe is Your Safety Metric? Automatic Concatenation Tests for Metric Reliability
This addresses a critical reliability issue in AI safety evaluation for developers and users, highlighting vulnerabilities in current metrics.
The paper tackled the problem of safety metrics for large language models being unreliable when harmful prompt-response pairs are concatenated, as they incorrectly label combined content as safe, and found that multiple advanced metrics exhibit this behavior with significant inconsistencies.
Consider a scenario where a harmfulness evaluation metric intended to filter unsafe responses from a Large Language Model. When applied to individual harmful prompt-response pairs, it correctly flags them as unsafe by assigning a high-risk score. Yet, if those same pairs are concatenated, the metrics decision unexpectedly reverses - labelling the combined content as safe with a low score, allowing the harmful text to bypass the filter. We found that multiple safety metrics, including advanced metrics such as GPT-based judges, exhibit this non-safe behaviour. Moreover, they show a strong sensitivity to input order: responses are often classified as safe if safe content appears first, regardless of any harmful content that follows, and vice versa. These findings underscore the importance of evaluating the safety of safety metrics, that is, the reliability of their output scores. To address this, we developed general, automatic, concatenation-based tests to assess key properties of these metrics. When applied in a model safety scenario, the tests revealed significant inconsistencies in harmfulness evaluations.