iRNN: Integer-only Recurrent Neural Network
This enables efficient deployment of RNNs for text and speech applications on edge devices like smartphones, addressing a specific bottleneck in AI deployment.
The authors tackled the problem of deploying recurrent neural networks (RNNs) with complex modules like layer normalization and attention on integer-only arithmetic, which was previously an open issue, by proposing a quantization-aware training method called iRNN. The result is an integer-only RNN that maintains similar performance to full-precision versions while improving runtime by 2× and reducing model size by 4× on smartphones.
Recurrent neural networks (RNN) are used in many real-world text and speech applications. They include complex modules such as recurrence, exponential-based activation, gate interaction, unfoldable normalization, bi-directional dependence, and attention. The interaction between these elements prevents running them on integer-only operations without a significant performance drop. Deploying RNNs that include layer normalization and attention on integer-only arithmetic is still an open problem. We present a quantization-aware training method for obtaining a highly accurate integer-only recurrent neural network (iRNN). Our approach supports layer normalization, attention, and an adaptive piecewise linear approximation of activations (PWL), to serve a wide range of RNNs on various applications. The proposed method is proven to work on RNN-based language models and challenging automatic speech recognition, enabling AI applications on the edge. Our iRNN maintains similar performance as its full-precision counterpart, their deployment on smartphones improves the runtime performance by $2\times$, and reduces the model size by $4\times$.