HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessment
This addresses the problem of unreliable jailbreak effectiveness reporting for LLM safety researchers, though it is incremental as it builds on existing metrics and datasets.
The paper tackles the lack of systematic benchmarks for evaluating metrics and judges that assess the harmfulness of LLM outputs, introducing HarmMetric Eval as a comprehensive benchmark with a dataset and scoring mechanism. Their experiments reveal that conventional metrics like METEOR and ROUGE-1 outperform LLM-based judges in this task, challenging assumptions about LLM superiority.
The alignment of large language models (LLMs) with human values is critical for their safe deployment, yet jailbreak attacks can subvert this alignment to elicit harmful outputs from LLMs. In recent years, a proliferation of jailbreak attacks has emerged, accompanied by diverse metrics and judges to assess the harmfulness of the LLM outputs. However, the absence of a systematic benchmark to assess the quality and effectiveness of these metrics and judges undermines the credibility of the reported jailbreak effectiveness and other risks. To address this gap, we introduce HarmMetric Eval, a comprehensive benchmark designed to support both overall and fine-grained evaluation of harmfulness metrics and judges. Our benchmark includes a high-quality dataset of representative harmful prompts paired with diverse harmful and non-harmful model responses, alongside a flexible scoring mechanism compatible with various metrics and judges. With HarmMetric Eval, our extensive experiments uncover a surprising result: two conventional metrics--METEOR and ROUGE-1--outperform LLM-based judges in evaluating the harmfulness of model responses, challenging prevailing beliefs about LLMs' superiority in this domain. Our dataset is publicly available at https://huggingface.co/datasets/qusgo/HarmMetric_Eval, and the code is available at https://anonymous.4open.science/r/HarmMetric-Eval-4CBE.