If It's Good Enough for You, It's Good Enough for Me: Transferability of Audio Sufficiencies across Models
This work addresses the need for better interpretability and comparison of audio models, particularly for researchers in audio classification, though it is incremental in extending transferability concepts from other domains.
The paper tackles the problem of understanding how different audio classification models process information by proposing transferability analysis, which tests whether a minimal sufficient signal for one model is accepted by others, and finds that transferability rates vary by task, with music genre signals transferable about 26% of the time, while deepfake detection models show unique behaviors.
In order to gain fresh insights about the information processing characteristics of different audio classification models, we propose transferability analysis. Given a minimal, sufficient signal for a classification on a model $f$, transferability analysis asks whether other models accept this minimal signal as having the same classification as it did on $f$. We define what it means for a sufficient signal to be transferable and perform a large study over $3$ different classification tasks: music genre, emotion recognition and deepfake detection. We find that transferability rates vary depending on the task, with sufficient signals for music genre being transferable $\approx26\%$ of the time. The other tasks reveal much higher variance in transferability and reveal that some models, in particular on deepfake detection, have different transferability behavior. We call these models `flat-earther' models. We investigate deepfake audio in more depth, and show that transferability analysis also allows to us to discover information theoretic differences between the models which are not captured by the more familiar metrics of accuracy and precision.