Evaluation of Thematic Coherence in Microblogs
This work addresses the need for better evaluation methods for thematic clustering in microblogs, which is useful for tasks like opinion analysis, but it is incremental as it builds on existing metrics.
The paper tackled the problem of evaluating thematic coherence in microblog clusters by creating a corpus from three domains and time windows and comparing automated metrics, finding that text generation metrics (TGMs) were more reliable than surface-level or topic coherence metrics, with TGMs showing less sensitivity to time windows.
Collecting together microblogs representing opinions about the same topics within the same timeframe is useful to a number of different tasks and practitioners. A major question is how to evaluate the quality of such thematic clusters. Here we create a corpus of microblog clusters from three different domains and time windows and define the task of evaluating thematic coherence. We provide annotation guidelines and human annotations of thematic coherence by journalist experts. We subsequently investigate the efficacy of different automated evaluation metrics for the task. We consider a range of metrics including surface level metrics, ones for topic model coherence and text generation metrics (TGMs). While surface level metrics perform well, outperforming topic coherence metrics, they are not as consistent as TGMs. TGMs are more reliable than all other metrics considered for capturing thematic coherence in microblog clusters due to being less sensitive to the effect of time windows.