Multi-modal Automated Speech Scoring using Attention Fusion
This work addresses the problem of efficient and accurate speech scoring for language learners, though it appears incremental as it builds on existing multi-modal methods with attention mechanisms.
The study tackled automated assessment of non-native English speakers' spontaneous speech by proposing a multi-modal neural approach using attention fusion, which significantly improved performance compared to strong baselines.
In this study, we propose a novel multi-modal end-to-end neural approach for automated assessment of non-native English speakers' spontaneous speech using attention fusion. The pipeline employs Bi-directional Recurrent Convolutional Neural Networks and Bi-directional Long Short-Term Memory Neural Networks to encode acoustic and lexical cues from spectrograms and transcriptions, respectively. Attention fusion is performed on these learned predictive features to learn complex interactions between different modalities before final scoring. We compare our model with strong baselines and find combined attention to both lexical and acoustic cues significantly improves the overall performance of the system. Further, we present a qualitative and quantitative analysis of our model.