Full-Frequency Temporal Patching and Structured Masking for Enhanced Audio Classification
This work addresses computational bottlenecks and performance limitations in audio classification for researchers and practitioners, representing an incremental improvement over existing patching methods.
The paper tackles the inefficiency and disruption of continuous frequency patterns in audio classification models by proposing Full-Frequency Temporal Patching (FFTP) and SpecMask augmentation, resulting in up to +6.76 mAP improvement on AudioSet-18k, +8.46 accuracy on SpeechCommandsV2, and 83.26% computation reduction.
Transformers and State-Space Models (SSMs) have advanced audio classification by modeling spectrograms as sequences of patches. However, existing models such as the Audio Spectrogram Transformer (AST) and Audio Mamba (AuM) adopt square patching from computer vision, which disrupts continuous frequency patterns and produces an excessive number of patches, slowing training, and increasing computation. We propose Full-Frequency Temporal Patching (FFTP), a patching strategy that better matches the time-frequency asymmetry of spectrograms by spanning full frequency bands with localized temporal context, preserving harmonic structure, and significantly reducing patch count and computation. We also introduce SpecMask, a patch-aligned spectrogram augmentation that combines full-frequency and localized time-frequency masks under a fixed masking budget, enhancing temporal robustness while preserving spectral continuity. When applied on both AST and AuM, our patching method with SpecMask improves mAP by up to +6.76 on AudioSet-18k and accuracy by up to +8.46 on SpeechCommandsV2, while reducing computation by up to 83.26%, demonstrating both performance and efficiency gains.