Study of positional encoding approaches for Audio Spectrogram Transformers
This work addresses the need for more efficient training of audio classification models without reliance on large-scale pretraining, though it is incremental as it focuses on optimizing a specific component.
The paper tackled the problem of Audio Spectrogram Transformers requiring ImageNet pretraining to outperform CNNs by studying positional encoding variants, resulting in a model with conditional positional encodings that significantly improved performance on Audioset and ESC-50 datasets.
Transformers have revolutionized the world of deep learning, specially in the field of natural language processing. Recently, the Audio Spectrogram Transformer (AST) was proposed for audio classification, leading to state of the art results in several datasets. However, in order for ASTs to outperform CNNs, pretraining with ImageNet is needed. In this paper, we study one component of the AST, the positional encoding, and propose several variants to improve the performance of ASTs trained from scratch, without ImageNet pretraining. Our best model, which incorporates conditional positional encodings, significantly improves performance on Audioset and ESC-50 compared to the original AST.