Whisper-MLA: Reducing GPU Memory Consumption of ASR Models based on MHA2MLA Conversion
This addresses a memory bottleneck for long-form audio ASR applications, offering an incremental improvement with practical deployment benefits.
The paper tackles the high GPU memory consumption in Transformer-based Whisper ASR models by introducing Whisper-MLA, which incorporates Multi-Head Latent Attention to reduce the Key-Value cache size by up to 87.5% while maintaining competitive accuracy on the LibriSpeech benchmark.
The Transformer-based Whisper model has achieved state-of-the-art performance in Automatic Speech Recognition (ASR). However, its Multi-Head Attention (MHA) mechanism results in significant GPU memory consumption due to the linearly growing Key-Value (KV) cache usage, which is problematic for many applications especially with long-form audio. To address this, we introduce Whisper-MLA, a novel architecture that incorporates Multi-Head Latent Attention (MLA) into the Whisper model. Specifically, we adapt MLA for Whisper's absolute positional embeddings and systematically investigate its application across encoder self-attention, decoder self-attention, and cross-attention modules. Empirical results indicate that applying MLA exclusively to decoder self-attention yields the desired balance between performance and memory efficiency. Our proposed approach allows conversion of a pretrained Whisper model to Whisper-MLA with minimal fine-tuning. Extensive experiments on the LibriSpeech benchmark validate the effectiveness of this conversion, demonstrating that Whisper-MLA reduces the KV cache size by up to 87.5% while maintaining competitive accuracy.