Challenges in the Automatic Analysis of Students' Diagnostic Reasoning
This work addresses the time-consuming manual analysis of diagnostic reasoning for educational psychologists, but it is incremental as it focuses on establishing a corpus and evaluation framework rather than a novel solution.
The paper tackles the problem of automating the identification of epistemic activities in students' diagnostic reasoning to enable large-scale analysis and feedback, creating the first annotated corpus and revealing that current state-of-the-art recurrent neural networks fail to address key challenges such as span identification and activity distinction.
Diagnostic reasoning is a key component of many professions. To improve students' diagnostic reasoning skills, educational psychologists analyse and give feedback on epistemic activities used by these students while diagnosing, in particular, hypothesis generation, evidence generation, evidence evaluation, and drawing conclusions. However, this manual analysis is highly time-consuming. We aim to enable the large-scale adoption of diagnostic reasoning analysis and feedback by automating the epistemic activity identification. We create the first corpus for this task, comprising diagnostic reasoning self-explanations of students from two domains annotated with epistemic activities. Based on insights from the corpus creation and the task's characteristics, we discuss three challenges for the automatic identification of epistemic activities using AI methods: the correct identification of epistemic activity spans, the reliable distinction of similar epistemic activities, and the detection of overlapping epistemic activities. We propose a separate performance metric for each challenge and thus provide an evaluation framework for future research. Indeed, our evaluation of various state-of-the-art recurrent neural network architectures reveals that current techniques fail to address some of these challenges.