Topic Modelling of Empirical Text Corpora: Validity, Reliability, and Reproducibility in Comparison to Semantic Maps
This work addresses the reliability and reproducibility of topic models for empirical text analysis, highlighting limitations for semantic interpretation in applications like grant selection, but it is incremental as it compares existing methods on a specific dataset.
The study compared topic modeling (LDA) and factor analysis (PCA) on 6,638 case descriptions, finding that LDA is more sensitive to sample changes (e.g., document removal causes larger distortions than in PCA) but outperforms PCA in semantic coherence.
Using the 6,638 case descriptions of societal impact submitted for evaluation in the Research Excellence Framework (REF 2014), we replicate the topic model (Latent Dirichlet Allocation or LDA) made in this context and compare the results with factor-analytic results using a traditional word-document matrix (Principal Component Analysis or PCA). Removing a small fraction of documents from the sample, for example, has on average a much larger impact on LDA than on PCA-based models to the extent that the largest distortion in the case of PCA has less effect than the smallest distortion of LDA-based models. In terms of semantic coherence, however, LDA models outperform PCA-based models. The topic models inform us about the statistical properties of the document sets under study, but the results are statistical and should not be used for a semantic interpretation - for example, in grant selections and micro-decision making, or scholarly work-without follow-up using domain-specific semantic maps.