Ace-CEFR -- A Dataset for Automated Evaluation of the Linguistic Difficulty of Conversational Texts for LLM Applications
This addresses the unmet need for automated difficulty assessment in conversational texts, particularly for LLM training and filtering, though it is incremental as it builds on existing annotation and modeling approaches.
The authors tackled the problem of evaluating language difficulty in conversational texts for LLM applications by introducing Ace-CEFR, a dataset expert-annotated for text difficulty levels, and showed that models trained on it can measure difficulty more accurately than human experts with production-appropriate latency.
There is an unmet need to evaluate the language difficulty of short, conversational passages of text, particularly for training and filtering Large Language Models (LLMs). We introduce Ace-CEFR, a dataset of English conversational text passages expert-annotated with their corresponding level of text difficulty. We experiment with several models on Ace-CEFR, including Transformer-based models and LLMs. We show that models trained on Ace-CEFR can measure text difficulty more accurately than human experts and have latency appropriate to production environments. Finally, we release the Ace-CEFR dataset to the public for research and development.