FlexiAST: Flexibility is What AST Needs
This addresses a flexibility issue for researchers and practitioners using ASTs in audio tasks, but it is incremental as it modifies training without new architectural innovations.
The paper tackled the problem of Audio Spectrogram Transformers (AST) degrading in performance when evaluated with patch sizes different from training, proposing FlexiAST, a training procedure that enables AST models to work with various patch sizes at inference without architectural changes, achieving similar performance to standard AST models across different datasets for audio classification.
The objective of this work is to give patch-size flexibility to Audio Spectrogram Transformers (AST). Recent advancements in ASTs have shown superior performance in various audio-based tasks. However, the performance of standard ASTs degrades drastically when evaluated using different patch sizes from that used during training. As a result, AST models are typically re-trained to accommodate changes in patch sizes. To overcome this limitation, this paper proposes a training procedure to provide flexibility to standard AST models without architectural changes, allowing them to work with various patch sizes at the inference stage - FlexiAST. This proposed training approach simply utilizes random patch size selection and resizing of patch and positional embedding weights. Our experiments show that FlexiAST gives similar performance to standard AST models while maintaining its evaluation ability at various patch sizes on different datasets for audio classification tasks.