Topics as Entity Clusters: Entity-based Topics from Large Language Models and Graph Neural Networks
This work addresses the challenge of interpretable topic modeling for researchers and practitioners in natural language processing, though it appears incremental as it builds on existing entity-based methods.
The paper tackled the problem of neural topic modeling with conceptual entities by proposing a bimodal vector representation approach using large language models and graph neural networks, achieving better coherency metrics compared to state-of-the-art models, especially with graph-based embeddings.
Topic models aim to reveal latent structures within a corpus of text, typically through the use of term-frequency statistics over bag-of-words representations from documents. In recent years, conceptual entities -- interpretable, language-independent features linked to external knowledge resources -- have been used in place of word-level tokens, as words typically require extensive language processing with a minimal assurance of interpretability. However, current literature is limited when it comes to exploring purely entity-driven neural topic modeling. For instance, despite the advantages of using entities for eliciting thematic structure, it is unclear whether current techniques are compatible with these sparsely organised, information-dense conceptual units. In this work, we explore entity-based neural topic modeling and propose a novel topic clustering approach using bimodal vector representations of entities. Concretely, we extract these latent representations from large language models and graph neural networks trained on a knowledge base of symbolic relations, in order to derive the most salient aspects of these conceptual units. Analysis of coherency metrics confirms that our approach is better suited to working with entities in comparison to state-of-the-art models, particularly when using graph-based embeddings trained on a knowledge base.