Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection
This addresses the need for more reliable evaluation methods in ATS, particularly for researchers developing models to reduce misinformation, though it is incremental as it builds on existing error analysis frameworks.
The paper tackles the problem of evaluating errors in Automatic Text Simplification (ATS) by introducing a new taxonomy for error classification and a human-annotated test collection of simplified scientific texts, enabling better assessment and improvement of ATS models.
The general public often encounters complex texts but does not have the time or expertise to fully understand them, leading to the spread of misinformation. Automatic Text Simplification (ATS) helps make information more accessible, but its evaluation methods have not kept up with advances in text generation, especially with Large Language Models (LLMs). In particular, recent studies have shown that current ATS metrics do not correlate with the presence of errors. Manual inspections have further revealed a variety of errors, underscoring the need for a more nuanced evaluation framework, which is currently lacking. This resource paper addresses this gap by introducing a test collection for detecting and classifying errors in simplified texts. First, we propose a taxonomy of errors, with a formal focus on information distortion. Next, we introduce a parallel dataset of automatically simplified scientific texts. This dataset has been human-annotated with labels based on our proposed taxonomy. Finally, we analyze the quality of the dataset, and we study the performance of existing models to detect and classify errors from that taxonomy. These contributions give researchers the tools to better evaluate errors in ATS, develop more reliable models, and ultimately improve the quality of automatically simplified texts.