Empowering Computing Education Researchers Through LLM-Assisted Content Analysis
This method helps computing education researchers conduct larger-scale, more generalizable studies, though it is incremental as it builds on existing content analysis techniques.
The paper tackles the challenge of enabling computing education researchers to analyze large volumes of qualitative data without increasing their burden, by proposing LLM-assisted content analysis (LACA), which is demonstrated to be reproducible and rigorous on a computing education dataset.
Computing education research (CER) is often instigated by practitioners wanting to improve both their own and the wider discipline's teaching practice. However, the latter is often difficult as many researchers lack the colleagues, resources, or capacity to conduct research that is generalisable or rigorous enough to advance the discipline. As a result, research methods that enable sense-making with larger volumes of qualitative data, while not increasing the burden on the researcher, have significant potential within CER. In this discussion paper, we propose such a method for conducting rigorous analysis on large volumes of textual data, namely a variation of LLM-assisted content analysis (LACA). This method combines content analysis with the use of large language models, empowering researchers to conduct larger-scale research which they would otherwise not be able to perform. Using a computing education dataset, we illustrate how LACA could be applied in a reproducible and rigorous manner. We believe this method has potential in CER, enabling more generalisable findings from a wider range of research. This, together with the development of similar methods, can help to advance both the practice and research quality of the CER discipline.