An Effective Automated Speaking Assessment Approach to Mitigating Data Scarcity and Imbalanced Distribution
This work addresses data challenges in automated speaking assessment for language learners, representing an incremental advance with specific performance gains.
The paper tackled data scarcity and imbalanced proficiency levels in automated speaking assessment by proposing metric-based classification and loss reweighting with SSL embeddings, resulting in over 10% improvement in CEFR prediction accuracy on the ICNALE dataset.
Automated speaking assessment (ASA) typically involves automatic speech recognition (ASR) and hand-crafted feature extraction from the ASR transcript of a learner's speech. Recently, self-supervised learning (SSL) has shown stellar performance compared to traditional methods. However, SSL-based ASA systems are faced with at least three data-related challenges: limited annotated data, uneven distribution of learner proficiency levels and non-uniform score intervals between different CEFR proficiency levels. To address these challenges, we explore the use of two novel modeling strategies: metric-based classification and loss reweighting, leveraging distinct SSL-based embedding features. Extensive experimental results on the ICNALE benchmark dataset suggest that our approach can outperform existing strong baselines by a sizable margin, achieving a significant improvement of more than 10% in CEFR prediction accuracy.