Effects of Collaboration on the Performance of Interactive Theme Discovery Systems
This work addresses the problem of evaluating interactive theme discovery systems for qualitative researchers, but it is incremental as it focuses on developing a framework rather than a new method or broad breakthrough.
The authors tackled the lack of a unified evaluation framework for NLP-assisted qualitative data analysis tools by proposing one to study collaboration settings, finding significant differences in output consistency, cohesiveness, and correctness between synchronous and asynchronous collaboration across three tools.
NLP-assisted solutions have gained considerable traction to support qualitative data analysis. However, no unified evaluation framework exists which can account for the many different settings in which qualitative researchers may employ them. In this paper, we propose an evaluation framework to study the way collaboration settings may produce different outcomes across a variety of interactive systems. Specifically, we study the impact of synchronous vs. asynchronous collaboration using three different NLP-assisted qualitative research tools and present a comprehensive analysis of significant differences in the consistency, cohesiveness, and correctness of their outputs.