Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations
This addresses the problem of integrating generative AI into qualitative research for researchers, offering design recommendations to improve collaboration, though it is incremental as it builds on existing human-AI collaboration concepts.
The study investigated how qualitative researchers collaborate with ChatGPT for thematic analysis, finding it enhanced coding efficiency, aided data exploration, and provided quantitative insights, but raised concerns about trustworthiness, reliability, and contextual understanding.
Generative artificial intelligence (GenAI) offers promising potential for advancing human-AI collaboration in qualitative research. However, existing works focused on conventional machine-learning and pattern-based AI systems, and little is known about how researchers interact with GenAI in qualitative research. This work delves into researchers' perceptions of their collaboration with GenAI, specifically ChatGPT. Through a user study involving ten qualitative researchers, we found ChatGPT to be a valuable collaborator for thematic analysis, enhancing coding efficiency, aiding initial data exploration, offering granular quantitative insights, and assisting comprehension for non-native speakers and non-experts. Yet, concerns about its trustworthiness and accuracy, reliability and consistency, limited contextual understanding, and broader acceptance within the research community persist. We contribute five actionable design recommendations to foster effective human-AI collaboration. These include incorporating transparent explanatory mechanisms, enhancing interface and integration capabilities, prioritising contextual understanding and customisation, embedding human-AI feedback loops and iterative functionality, and strengthening trust through validation mechanisms.