DHP Benchmark: Are LLMs Good NLG Evaluators?
This work addresses the need for better evaluation of LLMs as NLG evaluators, which is important for researchers and practitioners in AI and NLP, though it is incremental as it builds on existing 'LLM-as-a-judge' paradigms.
The authors tackled the problem of evaluating how well large language models (LLMs) can assess natural language generation (NLG) quality by proposing the DHP benchmarking framework, which uses hierarchical perturbations and statistical tests to provide quantitative scores, and they benchmarked five LLM families across six datasets covering four NLG tasks.
Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks; this is often referred to as ``LLM-as-a-judge'' paradigm. However, the capabilities of LLMs in evaluating NLG quality remain underexplored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs. This framework leverages hierarchically perturbed text data and statistical tests to systematically measure the NLG evaluation capabilities of LLMs. We re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM families provides critical insight into their strengths and limitations as NLG evaluators. Our dataset is available at https://huggingface.co/datasets/YCWANGVINCE/DHP_Benchmark.