AudioMAE++: learning better masked audio representations with SwiGLU FFNs
This work addresses the need for better self-supervised audio representations for tasks like audio classification and speech processing, but it is incremental as it builds on existing MAE frameworks with architectural tweaks.
The authors tackled the problem of improving masked autoencoder representations for audio by integrating newer transformer advancements like SwiGLU FFNs, resulting in AudioMAE++ models that outperform existing MAE approaches on 10 downstream tasks and scale better with up to 4x more parameters.
Masked Autoencoders (MAEs) trained on audio spectrogram patches have emerged as a prominent approach for learning self-supervised audio representations. While several recent papers have evaluated key aspects of training MAEs on audio data, the majority of these approaches still leverage vanilla transformer building blocks, whereas the transformer community has seen steady integration of newer architectural advancements. In this work, we propose AudioMAE++, a revamped audio masked autoencoder with two such enhancements, namely macaron-style transformer blocks with gated linear units. When pretrained on the AudioSet dataset, the proposed AudioMAE++ models outperform existing MAE based approaches on 10 diverse downstream tasks, demonstrating excellent performance on audio classification and speech-based benchmarks. The proposed AudioMAE++ models also demonstrate excellent scaling characteristics, outperforming directly comparable standard MAE baselines with up to 4x more parameters.