Lightweight Convolutional Approaches to Reading Comprehension on SQuAD
This addresses the problem of slow training in reading comprehension models for NLP researchers, though it is incremental as it adapts existing convolutional techniques to a known task.
The paper tackled reading comprehension on SQuAD by exploring convolutional models as an alternative to recurrent neural networks, achieving a 0.6238 F1 score with an ensemble and a 25% performance gain over RNN baselines while training 6 times faster.
Current state-of-the-art reading comprehension models rely heavily on recurrent neural networks. We explored an entirely different approach to question answering: a convolutional model. By their nature, these convolutional models are fast to train and capture local dependencies well, though they can struggle with longer-range dependencies and thus require augmentation to achieve comparable performance to RNN-based models. We conducted over two dozen controlled experiments with convolutional models and various kernel/attention/regularization schemes to determine the precise performance gains of each strategy, while maintaining a focus on speed. We ultimately ensembled three models: crossconv (0.5398 dev F1), attnconv (0.5665), and maybeconv (0.5285). The ensembled model was able to achieve a 0.6238 F1 score using the official SQuAD evaluation script. Our individual convolutional model crossconv was able to exceed the performance of the RNN-plus-attention baseline by 25% while training 6 times faster.