An Exploration of Dropout with RNNs for Natural Language Inference
This work addresses the need for optimized dropout strategies in NLI models, though it is incremental as it builds on existing RNN and dropout techniques.
The paper tackled the problem of evaluating dropout regularization in RNN models for Natural Language Inference by proposing a novel RNN model and testing dropout at different layers and rates, achieving accuracies of 86.14% on SNLI and 77.05% on SciTail.
Dropout is a crucial regularization technique for the Recurrent Neural Network (RNN) models of Natural Language Inference (NLI). However, dropout has not been evaluated for the effectiveness at different layers and dropout rates in NLI models. In this paper, we propose a novel RNN model for NLI and empirically evaluate the effect of applying dropout at different layers in the model. We also investigate the impact of varying dropout rates at these layers. Our empirical evaluation on a large (Stanford Natural Language Inference (SNLI)) and a small (SciTail) dataset suggest that dropout at each feed-forward connection severely affects the model accuracy at increasing dropout rates. We also show that regularizing the embedding layer is efficient for SNLI whereas regularizing the recurrent layer improves the accuracy for SciTail. Our model achieved an accuracy 86.14% on the SNLI dataset and 77.05% on SciTail.