A Meta-Analysis of LLM Effects on Students across Qualification, Socialisation, and Subjectification
It addresses the problem of evaluating LLMs in education beyond narrow metrics for educators and HCI researchers, offering a purpose-level view that is incremental in reframing the assessment.
This paper conducted a meta-analysis of 133 studies to assess the impact of large language models (LLMs) on education, finding positive but uneven effects: strong in qualification (e.g., as tutors), variable in socialisation, and fragile in subjectification, with design being a key factor.
Large language models (LLMs) are increasingly positioned as solutions for education, yet evaluations often reduce their impact to narrow performance metrics. This paper reframes the question by asking "what kind of impact should LLMs have in education?" Drawing on Biesta's tripartite account of good education: qualification, socialisation, and subjectification, we present a meta-analysis of 133 experimental and quasi-experimental studies (k = 188). Overall, the impact of LLMs on student learning is positive but uneven. Strong effects emerge in qualification, particularly when LLMs function as tutors in sustained interventions. Socialisation outcomes appear more variable, concentrated in sustained, reflective interventions. Subjectification, linked to autonomy and learner development, remains fragile, with improvements confined to small-scale, long-term studies. This purpose-level view highlights design as the decisive factor: without scaffolds for participation and agency, LLMs privilege what is easiest to measure while neglecting broader aims of education. For HCI and education, the issue is not just whether LLMs work, but what futures they enable or foreclose.