On the Effective Use of Pretraining for Natural Language Inference
This work addresses hyperparameter tuning for NLP researchers, but it is incremental as it builds on existing methods like GloVe and attention models.
The paper tackled the problem of optimizing pretrained word embeddings and weight initialization for natural language inference, finding that pretrained embeddings outperform random and retrofitted ones, and orthogonal initialization yields gains of up to 2.9%.
Neural networks have excelled at many NLP tasks, but there remain open questions about the performance of pretrained distributed word representations and their interaction with weight initialization and other hyperparameters. We address these questions empirically using attention-based sequence-to-sequence models for natural language inference (NLI). Specifically, we compare three types of embeddings: random, pretrained (GloVe, word2vec), and retrofitted (pretrained plus WordNet information). We show that pretrained embeddings outperform both random and retrofitted ones in a large NLI corpus. Further experiments on more controlled data sets shed light on the contexts for which retrofitted embeddings can be useful. We also explore two principled approaches to initializing the rest of the model parameters, Gaussian and orthogonal, showing that the latter yields gains of up to 2.9% in the NLI task.