Can LLMs Recognize Toxicity? A Structured Investigation Framework and Toxicity Metric
This work addresses the challenge of robust toxicity detection for LLM developers and users, but it is incremental as it builds on existing metrics with a novel method.
The paper tackles the problem of detecting toxicity in text generated by Large Language Models (LLMs) by introducing a new metric based on LLMs to flexibly measure toxicity according to given definitions, achieving a 12-point improvement in F1 score over conventional metrics. It also finds that LLMs are unsuitable for toxicity evaluations in unverified factors due to upstream toxicity influences.
In the pursuit of developing Large Language Models (LLMs) that adhere to societal standards, it is imperative to detect the toxicity in the generated text. The majority of existing toxicity metrics rely on encoder models trained on specific toxicity datasets, which are susceptible to out-of-distribution (OOD) problems and depend on the dataset's definition of toxicity. In this paper, we introduce a robust metric grounded on LLMs to flexibly measure toxicity according to the given definition. We first analyze the toxicity factors, followed by an examination of the intrinsic toxic attributes of LLMs to ascertain their suitability as evaluators. Finally, we evaluate the performance of our metric with detailed analysis. Our empirical results demonstrate outstanding performance in measuring toxicity within verified factors, improving on conventional metrics by 12 points in the F1 score. Our findings also indicate that upstream toxicity significantly influences downstream metrics, suggesting that LLMs are unsuitable for toxicity evaluations within unverified factors.