Audiomer: A Convolutional Transformer For Keyword Spotting
This work addresses the problem of efficient and effective keyword spotting for speech processing applications, representing an incremental improvement by adapting existing transformer techniques to audio.
The authors tackled the challenge of applying transformers to audio tasks by introducing Audiomer, which combines 1D Residual Networks with Performer Attention to achieve state-of-the-art performance in keyword spotting using raw audio waveforms, outperforming previous methods while being computationally cheaper and parameter-efficient.
Transformers have seen an unprecedented rise in Natural Language Processing and Computer Vision tasks. However, in audio tasks, they are either infeasible to train due to extremely large sequence length of audio waveforms or incur a performance penalty when trained on Fourier-based features. In this work, we introduce an architecture, Audiomer, where we combine 1D Residual Networks with Performer Attention to achieve state-of-the-art performance in keyword spotting with raw audio waveforms, outperforming all previous methods while being computationally cheaper and parameter-efficient. Additionally, our model has practical advantages for speech processing, such as inference on arbitrarily long audio clips owing to the absence of positional encoding. The code is available at https://github.com/The-Learning-Machines/Audiomer-PyTorch.