Multimodal Attention Merging for Improved Speech Recognition and Audio Event Classification
This addresses resource constraints in speech and audio processing, offering a zero-shot method with incremental improvements through a learnable variant.
The paper tackles the problem of limited fine-tuning compute and labeled data for speech and audio models by introducing Multimodal Attention Merging (MAM), which transfers knowledge from text and image models to reduce Word Error Rate by up to 6.70% for speech recognition and classification error by 10.63% for audio event classification.
Training large foundation models using self-supervised objectives on unlabeled data, followed by fine-tuning on downstream tasks, has emerged as a standard procedure. Unfortunately, the efficacy of this approach is often constrained by both limited fine-tuning compute and scarcity in labeled downstream data. We introduce Multimodal Attention Merging (MAM), an attempt that facilitates direct knowledge transfer from attention matrices of models rooted in high resource modalities, text and images, to those in resource-constrained domains, speech and audio, employing a zero-shot paradigm. MAM reduces the relative Word Error Rate (WER) of an Automatic Speech Recognition (ASR) model by up to 6.70%, and relative classification error of an Audio Event Classification (AEC) model by 10.63%. In cases where some data/compute is available, we present Learnable-MAM, a data-driven approach to merging attention matrices, resulting in a further 2.90% relative reduction in WER for ASR and 18.42% relative reduction in AEC compared to fine-tuning.