PiC: A Phrase-in-Context Dataset for Phrase Understanding and Semantic Search
This work addresses a gap in phrase understanding for NLP researchers, though it is incremental as it builds on existing contextualized embedding methods.
The authors tackled the lack of a benchmark for contextualized phrase embeddings by introducing PiC, a dataset of ~28K noun phrases with Wikipedia contexts, which improved ranking models' accuracy and pushed span-selection models to near-human accuracy of 95% EM on semantic search tasks.
While contextualized word embeddings have been a de-facto standard, learning contextualized phrase embeddings is less explored and being hindered by the lack of a human-annotated benchmark that tests machine understanding of phrase semantics given a context sentence or paragraph (instead of phrases alone). To fill this gap, we propose PiC -- a dataset of ~28K of noun phrases accompanied by their contextual Wikipedia pages and a suite of three tasks for training and evaluating phrase embeddings. Training on PiC improves ranking models' accuracy and remarkably pushes span-selection (SS) models (i.e., predicting the start and end index of the target phrase) near-human accuracy, which is 95% Exact Match (EM) on semantic search given a query phrase and a passage. Interestingly, we find evidence that such impressive performance is because the SS models learn to better capture the common meaning of a phrase regardless of its actual context. SotA models perform poorly in distinguishing two senses of the same phrase in two contexts (~60% EM) and in estimating the similarity between two different phrases in the same context (~70% EM).