Learning Intrinsic Sparse Structures within Long Short-Term Memory
This work addresses the problem of deploying efficient RNNs on resource-limited devices and large-scale services, presenting an incremental improvement in model compression techniques.
The paper tackles model compression for Recurrent Neural Networks (RNNs) by learning Intrinsic Sparse Structures (ISS) within Long Short-Term Memory (LSTM) units to reduce sizes of basic structures while maintaining dimension consistency, achieving a 10.59x speedup without perplexity loss on a language modeling task and enabling a compact model with 2.69M weights for question answering.
Model compression is significant for the wide adoption of Recurrent Neural Networks (RNNs) in both user devices possessing limited resources and business clusters requiring quick responses to large-scale service requests. This work aims to learn structurally-sparse Long Short-Term Memory (LSTM) by reducing the sizes of basic structures within LSTM units, including input updates, gates, hidden states, cell states and outputs. Independently reducing the sizes of basic structures can result in inconsistent dimensions among them, and consequently, end up with invalid LSTM units. To overcome the problem, we propose Intrinsic Sparse Structures (ISS) in LSTMs. Removing a component of ISS will simultaneously decrease the sizes of all basic structures by one and thereby always maintain the dimension consistency. By learning ISS within LSTM units, the obtained LSTMs remain regular while having much smaller basic structures. Based on group Lasso regularization, our method achieves 10.59x speedup without losing any perplexity of a language modeling of Penn TreeBank dataset. It is also successfully evaluated through a compact model with only 2.69M weights for machine Question Answering of SQuAD dataset. Our approach is successfully extended to non- LSTM RNNs, like Recurrent Highway Networks (RHNs). Our source code is publicly available at https://github.com/wenwei202/iss-rnns