SingaKids: A Multilingual Multimodal Dialogic Tutor for Language Learning
This work addresses the problem of developing robust and kid-friendly AI tutors for language learning across multiple languages and cultural contexts, representing an incremental improvement through integration and optimization of existing components.
The authors tackled the challenge of creating a consistent and engaging multilingual multimodal tutor for language learning by introducing SingaKids, which integrates dense image captioning, multilingual dialogic interaction, speech understanding, and speech generation, and empirical studies with elementary school students showed it provides effective dialogic teaching across different performance levels.
The integration of generative artificial intelligence into educational applications has enhanced personalized and interactive learning experiences, and it shows strong potential to promote young learners language acquisition. However, it is still challenging to ensure consistent and robust performance across different languages and cultural contexts, and kids-friendly design requires simplified instructions, engaging interactions, and age-appropriate scaffolding to maintain motivation and optimize learning outcomes. In this work, we introduce SingaKids, a dialogic tutor designed to facilitate language learning through picture description tasks. Our system integrates dense image captioning, multilingual dialogic interaction, speech understanding, and engaging speech generation to create an immersive learning environment in four languages: English, Mandarin, Malay, and Tamil. We further improve the system through multilingual pre-training, task-specific tuning, and scaffolding optimization. Empirical studies with elementary school students demonstrate that SingaKids provides effective dialogic teaching, benefiting learners at different performance levels.