Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classification
This work addresses the evaluation bottleneck for researchers in audio self-supervised learning, offering a more efficient paradigm that could reduce reliance on costly fine-tuning.
The paper tackled the problem of evaluating self-supervised audio models by identifying that global pooling creates an information bottleneck in linear probes, causing misrepresentation of embedding quality for multi-label audio classification. The result was that their binarized prototypical probes method outperformed linear and attentive probing across 13 datasets and 6 encoders, establishing probing as a competitive and efficient alternative to fine-tuning.
Although probing frozen models has become a standard evaluation paradigm, self-supervised learning in audio defaults to fine-tuning. A key reason is that global pooling creates an information bottleneck causing linear probes to misrepresent the embedding quality: The $\texttt{cls}$-token discards crucial token information about dispersed, localized events in multi-label audio. This weakness is rooted in the mismatch between the pretraining objective (operating globally) and the downstream task (localized events). Across a comprehensive benchmark of 13 datasets and 6 spectrogram-based encoders, we first investigate the global pooling bottleneck. We then introduce binarized prototypical probes: a lightweight and simple pooling method that learns prototypes to perform class-wise information aggregation. Despite its simplicity, our method notably outperforms linear and attentive probing. Our work establishes probing as a competitive and efficient paradigm for evaluating audio SSL models, challenging the reliance on costly fine-tuning.