Evaluating the Evaluators: Are readability metrics good measures of readability?
This addresses the need for reliable readability evaluation in PLS for non-expert audiences, though it is incremental as it builds on existing metrics and LMs.
The paper tackled the problem of evaluating readability in Plain Language Summarization (PLS) by showing that traditional metrics like Flesch-Kincaid Grade Level correlate poorly with human judgments, while Language Models achieve a Pearson correlation of 0.56 and better capture deeper readability aspects.
Plain Language Summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. In this paper, we conduct a thorough survey of PLS literature, and identify that the current standard practice for readability evaluation is to use traditional readability metrics, such as Flesch-Kincaid Grade Level (FKGL). However, despite proven utility in other fields, these metrics have not been compared to human readability judgments in PLS. We evaluate 8 readability metrics and show that most correlate poorly with human judgments, including the most popular metric, FKGL. We then show that Language Models (LMs) are better judges of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. Extending our analysis to PLS datasets, which contain summaries aimed at non-expert audiences, we find that LMs better capture deeper measures of readability, such as required background knowledge, and lead to different conclusions than the traditional metrics. Based on these findings, we offer recommendations for best practices in the evaluation of plain language summaries. We release our analysis code and survey data.