Controlling Difficulty of Generated Text for AI-Assisted Language Learning
This addresses the challenge of making AI-assisted language learning accessible to absolute beginners, though it is incremental as it builds on existing controllable generation methods.
The paper tackled the problem of adapting large language model outputs for beginner language learners by using controllable generation techniques, resulting in a significant improvement in output comprehensibility from 40.4% to 84.3% with future discriminators and introducing a novel evaluation metric, Token Miss Rate.
Practicing conversations with large language models (LLMs) presents a promising alternative to traditional in-person language learning. However, most LLMs generate text at a near-native level of complexity, making them ill-suited for beginner learners (CEFR: A1-A2). In this paper, we investigate whether controllable generation techniques -- specifically modular methods that do not require model fine-tuning -- can adapt LLM outputs to better support absolute beginners. We evaluate these methods through both automatic metrics and a user study with university-level learners of Japanese. Our findings show that while prompting alone fails to control output difficulty, the use of future discriminators (Yang and Klein, 2021) significantly improves output comprehensibility (from 40.4\% to 84.3\%). We further introduce a novel token-level evaluation metric, Token Miss Rate (TMR), that quantifies the proportion of incomprehensible tokens per utterance and correlates strongly with human judgments. To support future research in AI-assisted language learning, we release our code, models, annotation tools, and dataset.