Using Similarity Measures to Select Pretraining Data for NER
This work addresses the challenge of data selection for pretraining in NLP, specifically for NER, offering a method to improve efficiency and performance, but it is incremental as it builds on existing pretraining techniques.
The paper tackled the problem of selecting pretraining data for Named Entity Recognition (NER) by proposing three cost-effective similarity measures to quantify the relationship between source and target data, demonstrating that these measures predict model usefulness across 30 data pairs and showing that pretrained language models are more effective and predictable than word vectors, except when data is dissimilar.
Word vectors and Language Models (LMs) pretrained on a large amount of unlabelled data can dramatically improve various Natural Language Processing (NLP) tasks. However, the measure and impact of similarity between pretraining data and target task data are left to intuition. We propose three cost-effective measures to quantify different aspects of similarity between source pretraining and target task data. We demonstrate that these measures are good predictors of the usefulness of pretrained models for Named Entity Recognition (NER) over 30 data pairs. Results also suggest that pretrained LMs are more effective and more predictable than pretrained word vectors, but pretrained word vectors are better when pretraining data is dissimilar.