Fast Speech Foundation Model Distillation Using Interleaved Stacking
For practitioners deploying speech models in low-resource environments, this work offers a faster distillation training method without sacrificing quality.
The paper tackles the underexplored training efficiency of speech foundation model distillation, proposing interleaved stacking to accelerate training while preserving performance. On SUPERB, the method achieves comparable or better results than baseline distillation with up to 2x faster training.
Distilling a large speech foundation model (SFM) into an efficient student model has been successfully applied to low-resource environments. Although distillation reduces inference latency, it requires an additional student model training. However, the training efficiency of SFM distillation remains underexplored. In this work, we explore training acceleration of SFM distillation to speed up model deployment. We examine the potential of stacking, in which the model depth is progressively increased through training until the target model depth is reached. While existing stacking methods improve training speed, they suffer from performance degradation. To handle this limitation, we propose interleaved stacking, a novel stacking method that consistently preserves layer position throughout the stacking process. This property is particularly critical in SFMs, in which each layer encodes distinct layer-specific knowledge. We validate the effectiveness of the proposed method on SUPERB.