User Ex Machina : Simulation as a Design Probe in Human-in-the-Loop Text Analytics
This work highlights a critical issue for domain experts using topic models, as their interventions often degrade model quality in an unobservable way. This is an incremental finding for human-in-the-loop systems.
This paper investigates the sensitivity of topic models to user interactions in human-in-the-loop text analytics. It finds that user actions, even minor ones, can significantly and often negatively impact model quality, making it difficult for users to assess the changes.
Topic models are widely used analysis techniques for clustering documents and surfacing thematic elements of text corpora. These models remain challenging to optimize and often require a "human-in-the-loop" approach where domain experts use their knowledge to steer and adjust. However, the fragility, incompleteness, and opacity of these models means even minor changes could induce large and potentially undesirable changes in resulting model. In this paper we conduct a simulation-based analysis of human-centered interactions with topic models, with the objective of measuring the sensitivity of topic models to common classes of user actions. We find that user interactions have impacts that differ in magnitude but often negatively affect the quality of the resulting modelling in a way that can be difficult for the user to evaluate. We suggest the incorporation of sensitivity and "multiverse" analyses to topic model interfaces to surface and overcome these deficiencies.