Scaling up the think-aloud method
This work addresses the labor-intensive bottleneck in cognitive science for researchers studying human reasoning, though it is incremental as it applies existing NLP methods to a new domain.
The authors tackled the problem of scaling up the think-aloud method by automating transcription and annotation using NLP tools, enabling analysis of 640 participants' verbal reports in a mathematical reasoning task with moderate inter-rater reliability.
The think-aloud method, where participants voice their thoughts as they solve a task, is a valuable source of rich data about human reasoning processes. Yet, it has declined in popularity in contemporary cognitive science, largely because labor-intensive transcription and annotation preclude large sample sizes. Here, we develop methods to automate the transcription and annotation of verbal reports of reasoning using natural language processing tools, allowing for large-scale analysis of think-aloud data. In our study, 640 participants thought aloud while playing the Game of 24, a mathematical reasoning task. We automatically transcribed the recordings and coded the transcripts as search graphs, finding moderate inter-rater reliability with humans. We analyze these graphs and characterize consistency and variation in human reasoning traces. Our work demonstrates the value of think-aloud data at scale and serves as a proof of concept for the automated analysis of verbal reports.