Style-transfer and Paraphrase: Looking for a Sensible Semantic Similarity Metric
This work addresses the need for better semantic similarity metrics in NLP tasks such as style transfer and paraphrase, but it is incremental as it primarily evaluates existing methods without introducing a new one.
The paper analyzed over a dozen semantic similarity metrics for NLP tasks like style transfer and paraphrase, using a new dataset of 14,000 human-labeled sentence pairs, and found that none closely matched human judgment, with Word Mover Distance identified as the most reasonable current solution.
The rapid development of such natural language processing tasks as style transfer, paraphrase, and machine translation often calls for the use of semantic similarity metrics. In recent years a lot of methods to measure the semantic similarity of two short texts were developed. This paper provides a comprehensive analysis for more than a dozen of such methods. Using a new dataset of fourteen thousand sentence pairs human-labeled according to their semantic similarity, we demonstrate that none of the metrics widely used in the literature is close enough to human judgment in these tasks. A number of recently proposed metrics provide comparable results, yet Word Mover Distance is shown to be the most reasonable solution to measure semantic similarity in reformulated texts at the moment.