On Generalization Bounds of a Family of Recurrent Neural Networks
This work addresses the lack of theoretical foundations for RNNs, which are widely used in sequential data analysis, by providing new generalization bounds, though it is incremental in extending existing theory to specific variants.
The paper tackles the problem of missing theoretical understanding for recurrent neural networks (RNNs) by studying generalization bounds for vanilla RNNs and variants like MGU, LSTM, and Conv RNNs under the PAC-Learning framework, resulting in a significantly tighter bound for vanilla RNNs and the first bounds for the variants.
Recurrent Neural Networks (RNNs) have been widely applied to sequential data analysis. Due to their complicated modeling structures, however, the theory behind is still largely missing. To connect theory and practice, we study the generalization properties of vanilla RNNs as well as their variants, including Minimal Gated Unit (MGU), Long Short Term Memory (LSTM), and Convolutional (Conv) RNNs. Specifically, our theory is established under the PAC-Learning framework. The generalization bound is presented in terms of the spectral norms of the weight matrices and the total number of parameters. We also establish refined generalization bounds with additional norm assumptions, and draw a comparison among these bounds. We remark: (1) Our generalization bound for vanilla RNNs is significantly tighter than the best of existing results; (2) We are not aware of any other generalization bounds for MGU, LSTM, and Conv RNNs in the exiting literature; (3) We demonstrate the advantages of these variants in generalization.