Automated Coding of Communications in Collaborative Problem-solving Tasks Using ChatGPT
This work addresses the scalability challenge in assessing 21st-century skills like collaborative problem-solving for researchers and practitioners, though it is incremental as it adapts existing AI models to a specific domain.
The study tackled the bottleneck of manually coding communication data for collaborative problem-solving assessments by testing ChatGPT's ability to automate this process across five datasets and two frameworks, finding it achieves satisfactory performance with variations based on model, framework, and task characteristics, and that prompt refinement can sometimes improve accuracy.
Collaborative problem solving (CPS) is widely recognized as a critical 21st-century skill. Assessing CPS depends heavily on coding the communication data using a construct-relevant framework, and this process has long been a major bottleneck to scaling up such assessments. Based on five datasets and two coding frameworks, we demonstrate that ChatGPT can code communication data to a satisfactory level, though performance varies across ChatGPT models, and depends on the coding framework and task characteristics. Interestingly, newer reasoning-focused models such as GPT-o1-mini and GPT-o3-mini do not necessarily yield better coding results. Additionally, we show that refining prompts based on feedback from miscoded cases can improve coding accuracy in some instances, though the effectiveness of this approach is not consistent across all tasks. These findings offer practical guidance for researchers and practitioners in developing scalable, efficient methods to analyze communication data in support of 21st-century skill assessment.