SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity
This resource addresses the problem of robust evaluation and method development for verb semantics in NLP, though it is incremental as it builds on existing benchmarks by scaling up coverage and size.
The paper tackles the lack of attention to verbs in distributional semantics by introducing SimVerb-3500, a large-scale evaluation set with human similarity ratings for 3,500 verb pairs, covering all normed verb types from the USF database and providing at least three examples per VerbNet class.
Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research. We introduce SimVerb-3500, an evaluation resource that provides human ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed verb types from the USF free-association database, providing at least three examples for every VerbNet class. This broad coverage facilitates detailed analyses of how syntactic and semantic phenomena together influence human understanding of verb meaning. Further, with significantly larger development and test sets than existing benchmarks, SimVerb-3500 enables more robust evaluation of representation learning architectures and promotes the development of methods tailored to verbs. We hope that SimVerb-3500 will enable a richer understanding of the diversity and complexity of verb semantics and guide the development of systems that can effectively represent and interpret this meaning.