Optimizing Speech Multi-View Feature Fusion through Conditional Computation
This work addresses feature fusion challenges in speech translation, offering an incremental improvement for researchers and practitioners in speech processing.
The paper tackles the conflict between self-supervised learning (SSL) and traditional spectral features in speech translation by proposing a conditional computation framework with gradient-sensitive gating and multi-stage dropout, which accelerates convergence while maintaining performance comparable to spectral models on the MUSTC dataset.
Recent advancements have highlighted the efficacy of self-supervised learning (SSL) features in various speech-related tasks, providing lightweight and versatile multi-view speech representations. However, our study reveals that while SSL features expedite model convergence, they conflict with traditional spectral features like FBanks in terms of update directions. In response, we propose a novel generalized feature fusion framework grounded in conditional computation, featuring a gradient-sensitive gating network and a multi-stage dropout strategy. This framework mitigates feature conflicts and bolsters model robustness to multi-view input features. By integrating SSL and spectral features, our approach accelerates convergence and maintains performance on par with spectral models across multiple speech translation tasks on the MUSTC dataset.