Position: Thematic Analysis of Unstructured Clinical Transcripts with Large Language Models
This work tackles the problem of resource-intensive thematic analysis for researchers and clinicians, but it is incremental as it focuses on evaluation standardization rather than new methods or data.
The paper addresses the fragmented application of large language models (LLMs) to thematic analysis of clinical transcripts by reviewing studies and interviewing a clinician, finding that inconsistent evaluation methods hinder progress, and proposes a standardized framework based on validity, reliability, and interpretability.
This position paper examines how large language models (LLMs) can support thematic analysis of unstructured clinical transcripts, a widely used but resource-intensive method for uncovering patterns in patient and provider narratives. We conducted a systematic review of recent studies applying LLMs to thematic analysis, complemented by an interview with a practicing clinician. Our findings reveal that current approaches remain fragmented across multiple dimensions including types of thematic analysis, datasets, prompting strategies and models used, most notably in evaluation. Existing evaluation methods vary widely (from qualitative expert review to automatic similarity metrics), hindering progress and preventing meaningful benchmarking across studies. We argue that establishing standardized evaluation practices is critical for advancing the field. To this end, we propose an evaluation framework centered on three dimensions: validity, reliability, and interpretability.