Unifying Model and Layer Fusion for Speech Foundation Models
This work addresses the challenge of effectively utilizing Speech Foundation Models for downstream tasks, offering a promising approach for researchers and practitioners in speech processing, though it is incremental as it builds on existing fusion strategies.
The paper tackled the problem of improving performance on speech tasks by unifying model and layer fusion strategies, proposing an interface module that enables fusion across multiple upstream speech models and their layers, and demonstrated that this method outperforms prior fusion approaches on tasks like ASR and paralinguistic analysis.
Speech Foundation Models have gained significant attention recently. Prior works have shown that the fusion of representations from multiple layers of the same model or the fusion of multiple models can improve performance on downstream tasks. We unify these two fusion strategies by proposing an interface module that enables fusion across multiple upstream speech models while integrating information across their layers. We conduct extensive experiments on different self-supervised and supervised models across various speech tasks, including ASR and paralinguistic analysis, and demonstrate that our method outperforms prior fusion approaches. We further analyze its scalability concerning model size and count, highlighting the importance of selecting appropriate upstream models. Our results show that the proposed interface provides an additional performance boost when given a suitable upstream model selection, making it a promising approach for utilizing Speech Foundation Models.