Diagnostic Assessment Generation via Combinatorial Search
This work addresses the time-consuming task of creating diagnostic tests for educators, though it appears incremental as it builds on existing combinatorial search and genetic algorithm techniques.
The paper tackles the problem of automatically generating diagnostic assessment tests from problem-solving history by formulating question assembly as a combinatorial search over a learner-question knowledge matrix. The proposed genetic algorithm method outperforms greedy and random baselines by a large margin on one private and four public datasets, with qualitative analysis showing good problem scatterness and difficulty distribution for 9th graders.
Initial assessment tests are crucial in capturing learner knowledge states in a consistent manner. Aside from crafting questions itself, putting together relevant problems to form a question sheet is also a time-consuming process. In this work, we present a generic formulation of question assembly and a genetic algorithm based method that can generate assessment tests from raw problem-solving history. First, we estimate the learner-question knowledge matrix (snapshot). Each matrix element stands for the probability that a learner correctly answers a specific question. We formulate the task as a combinatorial search over this snapshot. To ensure representative and discriminative diagnostic tests, questions are selected (1) that has a low root mean squared error against the whole question pool and (2) high standard deviation among learner performances. Experimental results show that the proposed method outperforms greedy and random baseline by a large margin in one private dataset and four public datasets. We also performed qualitative analysis on the generated assessment test for 9th graders, which enjoys good problem scatterness across the whole 9th grader curriculum and decent difficulty level distribution.