Quantifying the Dissimilarity of Texts
This work addresses the need for effective dissimilarity measures in NLP tasks like retrieval and clustering, but it is incremental as it compares existing methods without introducing new ones.
The paper tackled the problem of quantifying text dissimilarity by comparing various measures and representations across different tasks, finding that the generalized Jensen-Shannon divergence on word frequencies performed strongly overall, while embedding-based methods excelled for smaller texts, with task-dependent optimal choices. It also demonstrated inconsistencies in Jaccard distance estimators and robustness of Jensen-Shannon divergence and embeddings to text length variations.
Quantifying the dissimilarity of two texts is an important aspect of a number of natural language processing tasks, including semantic information retrieval, topic classification, and document clustering. In this paper, we compared the properties and performance of different dissimilarity measures $D$ using three different representations of texts -- vocabularies, word frequency distributions, and vector embeddings -- and three simple tasks -- clustering texts by author, subject, and time period. Using the Project Gutenberg database, we found that the generalised Jensen--Shannon divergence applied to word frequencies performed strongly across all tasks, that $D$'s based on vector embedding representations led to stronger performance for smaller texts, and that the optimal choice of approach was ultimately task-dependent. We also investigated, both analytically and numerically, the behaviour of the different $D$'s when the two texts varied in length by a factor $h$. We demonstrated that the (natural) estimator of the Jaccard distance between vocabularies was inconsistent and computed explicitly the $h$-dependency of the bias of the estimator of the generalised Jensen--Shannon divergence applied to word frequencies. We also found numerically that the Jensen--Shannon divergence and embedding-based approaches were robust to changes in $h$, while the Jaccard distance was not.