GhostRNN: Reducing State Redundancy in RNN with Cheap Operations
This work addresses efficiency challenges for RNNs in speech applications on low-resource devices, presenting an incremental improvement.
The paper tackles the problem of hidden state redundancy in RNNs for low-resource devices by proposing GhostRNN, which reduces memory usage by ~40% and computation cost while maintaining similar performance on keyword spotting and speech enhancement tasks.
Recurrent neural network (RNNs) that are capable of modeling long-distance dependencies are widely used in various speech tasks, eg., keyword spotting (KWS) and speech enhancement (SE). Due to the limitation of power and memory in low-resource devices, efficient RNN models are urgently required for real-world applications. In this paper, we propose an efficient RNN architecture, GhostRNN, which reduces hidden state redundancy with cheap operations. In particular, we observe that partial dimensions of hidden states are similar to the others in trained RNN models, suggesting that redundancy exists in specific RNNs. To reduce the redundancy and hence computational cost, we propose to first generate a few intrinsic states, and then apply cheap operations to produce ghost states based on the intrinsic states. Experiments on KWS and SE tasks demonstrate that the proposed GhostRNN significantly reduces the memory usage (~40%) and computation cost while keeping performance similar.