BAT: Better Audio Transformer Guided by Convex Gated Probing
This addresses the need for reliable and reproducible audio SSL evaluation and improvement, though it appears incremental as it refines existing methods.
The paper tackled the problem that fine-tuning misrepresents audio self-supervised learning (SSL) embeddings, and introduced Convex Gated Probing (CGP) to close the gap between fine-tuning and probing, leading to the Better Audio Transformer (BAT) which established new state-of-the-art results on audio benchmarks.
Probing is widely adopted in computer vision to faithfully evaluate self-supervised learning (SSL) embeddings, as fine-tuning may misrepresent their inherent quality. In contrast, audio SSL models still rely on fine-tuning because simple probing fails to unlock their full potential and alters their rankings when competing for SOTA on AudioSet. Hence, a robust and efficient probing mechanism is required to guide the trajectory of audio SSL towards reliable and reproducible methods. We introduce Convex Gated Probing (CGP), a prototype-based method that drastically closes the gap between fine-tuning and probing in audio. CGP efficiently utilizes all frozen layers via a gating mechanism and exposes the location of latent task-relevant information. Guided by CGP, we rework the entire SSL pipeline of current SOTA audio models that use legacy implementations of prior SSL methods. By refining data preprocessing, model architecture, and pre-training recipe, we introduce Better Audio Transformer (BAT), and establish new SOTA on audio benchmarks.