Revisiting Automated Topic Model Evaluation with Large Language Models
This addresses a longstanding challenge in natural language processing for researchers and practitioners using topic models, though it appears incremental as it applies existing LLMs to a known bottleneck.
The paper tackles the problem of automatically evaluating topic model output and determining the optimal number of topics, finding that large language models correlate more strongly with human judgments than existing metrics and can return reasonable values for optimal topic numbers.
Topic models are used to make sense of large text collections. However, automatically evaluating topic model output and determining the optimal number of topics both have been longstanding challenges, with no effective automated solutions to date. This paper proposes using large language models to evaluate such output. We find that large language models appropriately assess the resulting topics, correlating more strongly with human judgments than existing automated metrics. We then investigate whether we can use large language models to automatically determine the optimal number of topics. We automatically assign labels to documents and choosing configurations with the most pure labels returns reasonable values for the optimal number of topics.