Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiency
This work addresses efficiency challenges for audio processing applications, but it is incremental as it builds on existing methods for speech transformers and quantization.
The paper tackles the problem of improving inference efficiency in audio language models by comparing transformer encoders with mixed convolutional and self-attention modules versus using self-attention alone, finding that a simpler self-attention approach achieves comparable efficiency, especially when combined with low-bit weight quantization to reduce error propagation.
In this paper, we show that a simple self-supervised pre-trained audio model can achieve comparable inference efficiency to more complicated pre-trained models with speech transformer encoders. These speech transformers rely on mixing convolutional modules with self-attention modules. They achieve state-of-the-art performance on ASR with top efficiency. We first show that employing these speech transformers as an encoder significantly improves the efficiency of pre-trained audio models as well. However, our study shows that we can achieve comparable efficiency with advanced self-attention solely. We demonstrate that this simpler approach is particularly beneficial with a low-bit weight quantization technique of a neural network to improve efficiency. We hypothesize that it prevents propagating the errors between different quantized modules compared to recent speech transformers mixing quantized convolution and the quantized self-attention modules.