Multi-channel Multi-frame ADL-MVDR for Target Speech Separation
This work addresses the problem of nonlinear distortions and residual noise in speech separation for improving automatic speech recognition systems.
This paper proposes a multi-channel multi-frame all deep learning (ADL)-MVDR approach for target speech separation, which aims to reduce nonlinear distortions harmful to ASR systems and address high levels of residual noise in conventional MVDR systems. Evaluated on a Mandarin audio-visual corpus, the proposed system outperforms state-of-the-art approaches across several objective evaluation metrics, including ASR performance.
Many purely neural network based speech separation approaches have been proposed to improve objective assessment scores, but they often introduce nonlinear distortions that are harmful to modern automatic speech recognition (ASR) systems. Minimum variance distortionless response (MVDR) filters are often adopted to remove nonlinear distortions, however, conventional neural mask-based MVDR systems still result in relatively high levels of residual noise. Moreover, the matrix inverse involved in the MVDR solution is sometimes numerically unstable during joint training with neural networks. In this study, we propose a multi-channel multi-frame (MCMF) all deep learning (ADL)-MVDR approach for target speech separation, which extends our preliminary multi-channel ADL-MVDR approach. The proposed MCMF ADL-MVDR system addresses linear and nonlinear distortions. Spatio-temporal cross correlations are also fully utilized in the proposed approach. The proposed systems are evaluated using a Mandarin audio-visual corpus and are compared with several state-of-the-art approaches. Experimental results demonstrate the superiority of our proposed systems under different scenarios and across several objective evaluation metrics, including ASR performance.