BaldWhisper: Faster Whisper with Head Shearing and Layer Merging
This work addresses the problem of deploying efficient speech recognition models on edge devices for low-resource language communities, representing an incremental improvement over existing pruning methods.
The paper tackles the challenge of pruning large pre-trained transformers like Whisper for low-resource languages with limited data, such as Bambara with only 32 hours of speech-to-text data, by proposing a new pruning recipe that compresses embeddings and merges layers, resulting in a model that is 48% smaller, 2.15x faster, and retains 90% of the original performance.
Pruning large pre-trained transformers for low-resource languages is challenging, as it often requires massive retraining data to recover performance. For instance, Distill-Whisper prunes Whisper by 40% and retrains on 21,000 hours of speech, far beyond what is available for most languages. Can Whisper be made lighter and faster for edge devices in data-scarce settings? Focusing on Bambara with only 32h of speech-to-text data, we propose a new pruning recipe. Instead of vocabulary pruning, which is unsuitable due to frequent code-switching by Bambara speakers, we compress the embeddings with low-rank decomposition and feature distillation. Rather than removing layers, we merge them to limit performance loss. The final model preserves 90% of the original performance while being 48% smaller and 2.15x faster on a MacBook Air M1.