Front-End Adapter: Adapting Front-End Input of Speech based Self-Supervised Learning for Speech Recognition
This work addresses a specific technical bottleneck in adapting speech SSL models for practical applications, but it is incremental in nature.
The paper tackles the problem of front-end input inconsistency between pre-training and fine-tuning in speech self-supervised learning models, proposing a front-end adapter that enables compatibility between filterbank features and waveform-based models, with experiments showing effectiveness on speech recognition tasks.
Recent years have witnessed a boom in self-supervised learning (SSL) in various areas including speech processing. Speech based SSL models present promising performance in a range of speech related tasks. However, the training of SSL models is computationally expensive and a common practice is to fine-tune a released SSL model on the specific task. It is essential to use consistent front-end input during pre-training and fine-tuning. This consistency may introduce potential issues when the optimal front-end is not the same as that used in pre-training. In this paper, we propose a simple but effective front-end adapter to address this front-end discrepancy. By minimizing the distance between the outputs of different front-ends, the filterbank feature (Fbank) can be compatible with SSL models which are pre-trained with waveform. The experiment results demonstrate the effectiveness of our proposed front-end adapter on several popular SSL models for the speech recognition task.