On the choice of the optimal temporal support for audio classification with Pre-trained embeddings
This work addresses the often-overlooked aspect of audio input duration for practitioners using pre-trained embeddings in audio classification, though it is incremental as it focuses on optimizing existing methods rather than introducing new ones.
The study investigated how the duration of audio input (Temporal Support) affects the performance of pre-trained embeddings for audio classification, finding that Audio Spectrogram Transformer-based systems like PaSST and BEATs remain effective with smaller durations, reducing memory and computational costs, and achieved competitive results, including state-of-the-art improvements on OpenMIC without fine-tuning.
Current state-of-the-art audio analysis systems rely on pre-trained embedding models, often used off-the-shelf as (frozen) feature extractors. Choosing the best one for a set of tasks is the subject of many recent publications. However, one aspect often overlooked in these works is the influence of the duration of audio input considered to extract an embedding, which we refer to as Temporal Support (TS). In this work, we study the influence of the TS for well-established or emerging pre-trained embeddings, chosen to represent different types of architectures and learning paradigms. We conduct this evaluation using both musical instrument and environmental sound datasets, namely OpenMIC, TAU Urban Acoustic Scenes 2020 Mobile, and ESC-50. We especially highlight that Audio Spectrogram Transformer-based systems (PaSST and BEATs) remain effective with smaller TS, which therefore allows for a drastic reduction in memory and computational cost. Moreover, we show that by choosing the optimal TS we reach competitive results across all tasks. In particular, we improve the state-of-the-art results on OpenMIC, using BEATs and PaSST without any fine-tuning.