Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering
For practitioners deploying large speech models, this offers a practical compression technique that requires no data or training, achieving competitive performance with minimal overhead.
This work introduces a data-free and training-free compression method for speech foundation models using channelwise k-means clustering and mixed sparsity pruning. On HuBERT-large at 50% sparsity, it achieves up to 34.37%/21.91% relative WER reduction over magnitude-based pruning before fine-tuning, and after 3 epochs of fine-tuning, WER is within 3.36%/4.62% relative of the uncompressed baseline.
This paper presents a novel data-free and training-free compression approach for speech foundation models using channelwise clustering via k-means. More fine-grained, mixed sparsity pruning by layer-level varying number of parameter clusters is also explored. Experiments conducted on the LibriSpeech dataset suggest that when operating with pruning sparsity of 50% on HuBERT-large, consistent WER reductions of 27.73%/18.61% absolute (34.37%/21.91% relative) over the magnitude-based pruning were obtained on the test-clean and test-other subsets before fine-tuning and 0.19%/0.79% absolute (3.36%/4.62% relative) after fine-tuning with only 3 epochs. Similar WER reductions of 2.86%/5.02% absolute (59.21%/55.29% relative) were observed against magnitudebased pruning on Whisper-large-v3 at 10% sparsity, all with no significant WER increase relative to the uncompressed baseline.