Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts
This work addresses the limited research on reading evaluation systems, providing a tool to help teachers more efficiently score children's reading exercises, though it is incremental as it builds on existing models with prompting techniques.
The paper tackled the problem of automatic reading aloud evaluation for children by developing a multimodal approach using prompts with Whisper and LLMs, achieving a word error rate of 5.1% (improving from 9.4%) and increasing the F1 score for mistake detection from 0.39 to 0.73.
Automatic reading aloud evaluation can provide valuable support to teachers by enabling more efficient scoring of reading exercises. However, research on reading evaluation systems and applications remains limited. We present a novel multimodal approach that leverages audio and knowledge from text resources. In particular, we explored the potential of using Whisper and instruction-tuned large language models (LLMs) with prompts to improve transcriptions for child speech recognition, as well as their effectiveness in downstream reading mistake detection. Our results demonstrate the effectiveness of prompting Whisper and prompting LLM, compared to the baseline Whisper model without prompting. The best performing system achieved state-of-the-art recognition performance in Dutch child read speech, with a word error rate (WER) of 5.1%, improving the baseline WER of 9.4%. Furthermore, it significantly improved reading mistake detection, increasing the F1 score from 0.39 to 0.73.