Differentially Private Adapters for Parameter Efficient Acoustic Modeling
This work addresses privacy-preserving adaptation for speech modeling, offering a more efficient solution for domains like multilingual speech processing, though it is incremental as it builds on existing adaptation and DP techniques.
The paper tackles the problem of adapting cross-lingual speech classifiers with differential privacy guarantees by introducing a parameter-efficient method using residual adapters, reducing trainable parameters by 97.5% with only a 4% performance drop compared to fine-tuning.
In this work, we devise a parameter-efficient solution to bring differential privacy (DP) guarantees into adaptation of a cross-lingual speech classifier. We investigate a new frozen pre-trained adaptation framework for DP-preserving speech modeling without full model fine-tuning. First, we introduce a noisy teacher-student ensemble into a conventional adaptation scheme leveraging a frozen pre-trained acoustic model and attain superior performance than DP-based stochastic gradient descent (DPSGD). Next, we insert residual adapters (RA) between layers of the frozen pre-trained acoustic model. The RAs reduce training cost and time significantly with a negligible performance drop. Evaluated on the open-access Multilingual Spoken Words (MLSW) dataset, our solution reduces the number of trainable parameters by 97.5% using the RAs with only a 4% performance drop with respect to fine-tuning the cross-lingual speech classifier while preserving DP guarantees.