Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition
This work addresses deployment challenges for RNNs in sequence analysis by providing a more efficient compression method, though it is incremental as it builds on existing tensor decomposition approaches.
The paper tackles the problem of large model sizes in Recurrent Neural Networks (RNNs) for high-dimensional data by proposing Hierarchical Tucker (HT) decomposition for compression, resulting in HT-LSTM models that achieve higher compression ratios and test accuracy compared to state-of-the-art compressed RNNs like TT-LSTM, TR-LSTM, and BT-LSTM.
Recurrent Neural Networks (RNNs) have been widely used in sequence analysis and modeling. However, when processing high-dimensional data, RNNs typically require very large model sizes, thereby bringing a series of deployment challenges. Although the state-of-the-art tensor decomposition approaches can provide good model compression performance, these existing methods are still suffering some inherent limitations, such as restricted representation capability and insufficient model complexity reduction. To overcome these limitations, in this paper we propose to develop compact RNN models using Hierarchical Tucker (HT) decomposition. HT decomposition brings strong hierarchical structure to the decomposed RNN models, which is very useful and important for enhancing the representation capability. Meanwhile, HT decomposition provides higher storage and computational cost reduction than the existing tensor decomposition approaches for RNN compression. Our experimental results show that, compared with the state-of-the-art compressed RNN models, such as TT-LSTM, TR-LSTM and BT-LSTM, our proposed HT-based LSTM (HT-LSTM), consistently achieves simultaneous and significant increases in both compression ratio and test accuracy on different datasets.