Bad Form: Comparing Context-Based and Form-Based Few-Shot Learning in Distributional Semantic Models
This work addresses evaluation challenges in NLP for researchers, but it is incremental as it focuses on improving task design and hyperparameter tuning rather than introducing a new paradigm.
The paper tackled the problem of evaluating few-shot learning in distributional semantic models by showing that existing tasks are insufficient for comparing context-based and form-based methods, as they allow models to exploit word-form similarities in training and test data. They introduced three new tasks for balanced comparison and demonstrated that optimizing previously ignored hyperparameters improved performance, achieving state-of-the-art results on 4 out of 6 tasks.
Word embeddings are an essential component in a wide range of natural language processing applications. However, distributional semantic models are known to struggle when only a small number of context sentences are available. Several methods have been proposed to obtain higher-quality vectors for these words, leveraging both this context information and sometimes the word forms themselves through a hybrid approach. We show that the current tasks do not suffice to evaluate models that use word-form information, as such models can easily leverage word forms in the training data that are related to word forms in the test data. We introduce 3 new tasks, allowing for a more balanced comparison between models. Furthermore, we show that hyperparameters that have largely been ignored in previous work can consistently improve the performance of both baseline and advanced models, achieving a new state of the art on 4 out of 6 tasks.