Enhancing conversational quality in language learning chatbots: An evaluation of GPT4 for ASR error correction
This addresses the challenge of improving conversational quality for language learners using chatbots, though it is incremental as it applies an existing model to a specific domain.
This paper tackled the problem of high word-error-rate (WER) in recognizing non-native speech for language learning chatbots, which disrupts conversation flow, by using GPT4 for ASR error correction and found that it improves conversation quality despite increasing WER, outperforming standard methods without in-domain training data.
The integration of natural language processing (NLP) technologies into educational applications has shown promising results, particularly in the language learning domain. Recently, many spoken open-domain chatbots have been used as speaking partners, helping language learners improve their language skills. However, one of the significant challenges is the high word-error-rate (WER) when recognizing non-native/non-fluent speech, which interrupts conversation flow and leads to disappointment for learners. This paper explores the use of GPT4 for ASR error correction in conversational settings. In addition to WER, we propose to use semantic textual similarity (STS) and next response sensibility (NRS) metrics to evaluate the impact of error correction models on the quality of the conversation. We find that transcriptions corrected by GPT4 lead to higher conversation quality, despite an increase in WER. GPT4 also outperforms standard error correction methods without the need for in-domain training data.