Impact of Training Dataset Size on Neural Answer Selection Models
This addresses the problem of dataset efficiency for researchers and practitioners in NLP, but it is incremental as it builds on existing methods without introducing new techniques.
The study investigated how dataset size affects neural answer selection models, finding that some models showed little performance change with varying dataset sizes, questioning the algorithms' effectiveness.
It is held as a truism that deep neural networks require large datasets to train effective models. However, large datasets, especially with high-quality labels, can be expensive to obtain. This study sets out to investigate (i) how large a dataset must be to train well-performing models, and (ii) what impact can be shown from fractional changes to the dataset size. A practical method to investigate these questions is to train a collection of deep neural answer selection models using fractional subsets of varying sizes of an initial dataset. We observe that dataset size has a conspicuous lack of effect on the training of some of these models, bringing the underlying algorithms into question.