Evaluating Factuality in Text Simplification
This addresses the risk of providing inaccurate simplified texts, which could be harmful for users like lay readers accessing complex information such as medical literature, and is an incremental step by extending factuality concerns from summarization to simplification.
The paper tackles the problem of factual errors in automated text simplification models, which can introduce unsupported statements or omit key information, and finds that errors are prevalent in both reference datasets and state-of-the-art model outputs, highlighting gaps in current evaluation metrics.
Automated simplification models aim to make input texts more readable. Such methods have the potential to make complex information accessible to a wider audience, e.g., providing access to recent medical literature which might otherwise be impenetrable for a lay reader. However, such models risk introducing errors into automatically simplified texts, for instance by inserting statements unsupported by the corresponding original text, or by omitting key information. Providing more readable but inaccurate versions of texts may in many cases be worse than providing no such access at all. The problem of factual accuracy (and the lack thereof) has received heightened attention in the context of summarization models, but the factuality of automatically simplified texts has not been investigated. We introduce a taxonomy of errors that we use to analyze both references drawn from standard simplification datasets and state-of-the-art model outputs. We find that errors often appear in both that are not captured by existing evaluation metrics, motivating a need for research into ensuring the factual accuracy of automated simplification models.