A correlation-permutation approach for speech-music encoders model merging
This enables efficient creation of unified audio models for applications in speech and music processing, though it is incremental as it extends prior work to transformer layers.
The paper tackles the problem of merging independently trained speech and music encoders into a unified audio model without expensive retraining, achieving a 14.83-point improvement in music performance compared to linear interpolation merging.
Creating a unified speech and music model requires expensive pre-training. Model merging can instead create an unified audio model with minimal computational expense. However, direct merging is challenging when the models are not aligned in the weight space. Motivated by Git Re-Basin, we introduce a correlation-permutation approach that aligns a music encoder's internal layers with a speech encoder. We extend previous work to the case of merging transformer layers. The method computes a permutation matrix that maximizes the model's features-wise cross-correlations layer by layer, enabling effective fusion of these otherwise disjoint models. The merged model retains speech capabilities through this method while significantly enhancing music performance, achieving an improvement of 14.83 points in average score compared to linear interpolation model merging. This work allows the creation of unified audio models from independently trained encoders.