Approach to Learning Generalized Audio Representation Through Batch Embedding Covariance Regularization and Constant-Q Transforms
This work addresses the need for general-purpose audio embeddings in various application scenarios, but it appears incremental as it builds on existing models and benchmarks like HEAR 2021.
The paper tackled the problem of learning generalized audio representations for few-shot and zero-shot learning by proposing Batch Embedding Covariance Regularization (BECR) and testing different audio preprocessing methods like CQT and STFT. The results showed that BECR improves the PaSST model without extra computation, and STFT outperformed CQT in all tasks tested.
General-purpose embedding is highly desirable for few-shot even zero-shot learning in many application scenarios, including audio tasks. In order to understand representations better, we conducted a thorough error analysis and visualization of HEAR 2021 submission results. Inspired by the analysis, this work experiments with different front-end audio preprocessing methods, including Constant-Q Transform (CQT) and Short-time Fourier transform (STFT), and proposes a Batch Embedding Covariance Regularization (BECR) term to uncover a more holistic simulation of the frequency information received by the human auditory system. We tested the models on the suite of HEAR 2021 tasks, which encompass a broad category of tasks. Preliminary results show (1) the proposed BECR can incur a more dispersed embedding on the test set, (2) BECR improves the PaSST model without extra computation complexity, and (3) STFT preprocessing outperforms CQT in all tasks we tested. Github:https://github.com/ankitshah009/general_audio_embedding_hear_2021