Transfer Learning from Whisper for Microscopic Intelligibility Prediction
This work addresses the challenge of modeling detailed human speech perception for applications in hearing aids or speech technology, representing an incremental advance by extending transfer learning methods from macroscopic to microscopic intelligibility prediction.
The paper tackled the problem of predicting fine-grained listener perceptions (microscopic intelligibility) by applying transfer learning from the Whisper speech recognition model, achieving up to 66% relative improvement over baselines when fine-tuned.
Macroscopic intelligibility models predict the expected human word-error-rate for a given speech-in-noise stimulus. In contrast, microscopic intelligibility models aim to make fine-grained predictions about listeners' perception, e.g. predicting phonetic or lexical responses. State-of-the-art macroscopic models use transfer learning from large scale deep learning models for speech processing, whereas such methods have rarely been used for microscopic modeling. In this paper, we study the use of transfer learning from Whisper, a state-of-the-art deep learning model for automatic speech recognition, for microscopic intelligibility prediction at the level of lexical responses. Our method outperforms the considered baselines, even in a zero-shot setup, and yields a relative improvement of up to 66\% when fine-tuned to predict listeners' responses. Our results showcase the promise of large scale deep learning based methods for microscopic intelligibility prediction.