Audio Mamba: Pretrained Audio State Space Model For Audio Tagging
This addresses the scaling limitations in audio tagging models, offering a more parameter-efficient solution for researchers and practitioners in audio processing.
The paper tackles the quadratic self-attention cost in audio transformer models by proposing Audio Mamba, a self-attention-free state space model for audio tagging, achieving comparable results to state-of-the-art audio spectrogram transformers with one third the parameters.
Audio tagging is an important task of mapping audio samples to their corresponding categories. Recently endeavours that exploit transformer models in this field have achieved great success. However, the quadratic self-attention cost limits the scaling of audio transformer models and further constrains the development of more universal audio models. In this paper, we attempt to solve this problem by proposing Audio Mamba, a self-attention-free approach that captures long audio spectrogram dependency with state space models. Our experimental results on two audio-tagging datasets demonstrate the parameter efficiency of Audio Mamba, it achieves comparable results to SOTA audio spectrogram transformers with one third parameters.