Dropout Model Evaluation in MOOCs
This work addresses the need for more rigorous evaluation methods in learning analytics, with practical implications for designing interventions for at-risk students in MOOCs.
The authors tackled the problem of rigorous predictive model evaluation in learning analytics by introducing a statistical testing procedure for comparing algorithms and feature extraction methods. They applied this method to MOOCs and found that clickstream-based features significantly outperform forum- and assignment-based features for dropout prediction, with the latter two being indistinguishable.
The field of learning analytics needs to adopt a more rigorous approach for predictive model evaluation that matches the complex practice of model-building. In this work, we present a procedure to statistically test hypotheses about model performance which goes beyond the state-of-the-practice in the community to analyze both algorithms and feature extraction methods from raw data. We apply this method to a series of algorithms and feature sets derived from a large sample of Massive Open Online Courses (MOOCs). While a complete comparison of all potential modeling approaches is beyond the scope of this paper, we show that this approach reveals a large gap in dropout prediction performance between forum-, assignment-, and clickstream-based feature extraction methods, where the latter is significantly better than the former two, which are in turn indistinguishable from one another. This work has methodological implications for evaluating predictive or AI-based models of student success, and practical implications for the design and targeting of at-risk student models and interventions.