A Sensitivity Analysis of Attention-Gated Convolutional Neural Networks for Sentence Classification
This is an incremental study providing practical advice for researchers and practitioners using AGCNNs in natural language processing.
The paper tackles the problem of optimizing hyperparameters for Attention-Gated Convolutional Neural Networks (AGCNNs) in sentence classification, resulting in average improvements of 0.81% and 0.67% on certain variants and 0.47% and 0.45% on others.
In this paper, we investigate the effect of different hyperparameters as well as different combinations of hyperparameters settings on the performance of the Attention-Gated Convolutional Neural Networks (AGCNNs), e.g., the kernel window size, the number of feature maps, the keep rate of the dropout layer, and the activation function. We draw practical advice from a wide range of empirical results. Through the sensitivity analysis, we further improve the hyperparameters settings of AGCNNs. Experiments show that our proposals could achieve an average of 0.81% and 0.67% improvements on AGCNN-NLReLU-rand and AGCNN-SELU-rand, respectively; and an average of 0.47% and 0.45% improvements on AGCNN-NLReLU-static and AGCNN-SELU-static, respectively.