Thick-Net: Parallel Network Structure for Sequential Modeling
This addresses the challenge of enhancing sequential modeling for tasks such as text classification and language modeling, though it appears incremental as it builds on existing RNN structures.
The paper tackles the problem of improving recurrent neural networks for sequence learning by proposing Thick-Net, a model that expands networks in thickness with parallel parameters to avoid overfitting and ease optimization, resulting in improved accuracy, faster convergence, and better generalization across tasks like adding problem and language modeling.
Recurrent neural networks have been widely used in sequence learning tasks. In previous studies, the performance of the model has always been improved by either wider or deeper structures. However, the former becomes more prone to overfitting, while the latter is difficult to optimize. In this paper, we propose a simple new model named Thick-Net, by expanding the network from another dimension: thickness. Multiple parallel values are obtained via more sets of parameters in each hidden state, and the maximum value is selected as the final output among parallel intermediate outputs. Notably, Thick-Net can efficiently avoid overfitting, and is easier to optimize than the vanilla structures due to the large dropout affiliated with it. Our model is evaluated on four sequential tasks including adding problem, permuted sequential MNIST, text classification and language modeling. The results of these tasks demonstrate that our model can not only improve accuracy with faster convergence but also facilitate a better generalization ability.