Evaluating semantic models with word-sentence relatedness
This work addresses the need for better evaluation methods for semantic models in natural language processing, though it is incremental as it builds on existing datasets and tasks.
The authors tackled the problem of evaluating semantic textual similarity (STS) systems by developing a dataset of 775 English word-sentence pairs annotated for semantic relatedness by human raters, and found that while some off-the-shelf STS models captured much of the variance in human judgments, they were not sensitive to implicatures and entailments considered by participants.
Semantic textual similarity (STS) systems are designed to encode and evaluate the semantic similarity between words, phrases, sentences, and documents. One method for assessing the quality or authenticity of semantic information encoded in these systems is by comparison with human judgments. A data set for evaluating semantic models was developed consisting of 775 English word-sentence pairs, each annotated for semantic relatedness by human raters engaged in a Maximum Difference Scaling (MDS) task, as well as a faster alternative task. As a sample application of this relatedness data, behavior-based relatedness was compared to the relatedness computed via four off-the-shelf STS models: n-gram, Latent Semantic Analysis (LSA), Word2Vec, and UMBC Ebiquity. Some STS models captured much of the variance in the human judgments collected, but they were not sensitive to the implicatures and entailments that were processed and considered by the participants. All text stimuli and judgment data have been made freely available.