ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?
This research addresses the need for efficient and transparent evaluation of AI explanations, which is crucial for responsible AI development, though it is incremental in exploring existing methods on new data.
The study tackled the problem of evaluating natural language explanations (NLEs) by comparing ChatGPT's assessments to human judgments across binary, ternary, and 7-Likert scales, finding that ChatGPT aligns better with humans on coarser-grained scales and that paired comparisons and dynamic prompting improve alignment.
As AI becomes more integral in our lives, the need for transparency and responsibility grows. While natural language explanations (NLEs) are vital for clarifying the reasoning behind AI decisions, evaluating them through human judgments is complex and resource-intensive due to subjectivity and the need for fine-grained ratings. This study explores the alignment between ChatGPT and human assessments across multiple scales (i.e., binary, ternary, and 7-Likert scale). We sample 300 data instances from three NLE datasets and collect 900 human annotations for both informativeness and clarity scores as the text quality measurement. We further conduct paired comparison experiments under different ranges of subjectivity scores, where the baseline comes from 8,346 human annotations. Our results show that ChatGPT aligns better with humans in more coarse-grained scales. Also, paired comparisons and dynamic prompting (i.e., providing semantically similar examples in the prompt) improve the alignment. This research advances our understanding of large language models' capabilities to assess the text explanation quality in different configurations for responsible AI development.