Investigating the Sensitivity of Pre-trained Audio Embeddings to Common Effects
This work addresses the sensitivity of audio embeddings to effects for researchers and practitioners in audio processing, but it is incremental as it analyzes existing models without proposing new solutions.
The study investigated how pre-trained audio embeddings from models like OpenL3, PANNs, and CLAP respond to common audio effects such as gain and reverberation, finding that the embeddings move in high-dimensional subspaces and do not linearize the effects, with empirical tests showing no improvement in robustness for instrument classification tasks.
In recent years, foundation models have significantly advanced data-driven systems across various domains. Yet, their underlying properties, especially when functioning as feature extractors, remain under-explored. In this paper, we investigate the sensitivity to audio effects of audio embeddings extracted from widely-used foundation models, including OpenL3, PANNs, and CLAP. We focus on audio effects as the source of sensitivity due to their prevalent presence in large audio datasets. By applying parameterized audio effects (gain, low-pass filtering, reverberation, and bitcrushing), we analyze the correlation between the deformation trajectories and the effect strength in the embedding space. We propose to quantify the dimensionality and linearizability of the deformation trajectories induced by audio effects using canonical correlation analysis. We find that there exists a direction along which the embeddings move monotonically as the audio effect strength increases, but that the subspace containing the displacements is generally high-dimensional. This shows that pre-trained audio embeddings do not globally linearize the effects. Our empirical results on instrument classification downstream tasks confirm that projecting out the estimated deformation directions cannot generally improve the robustness of pre-trained embeddings to audio effects.