AI Meets the Classroom: When Do Large Language Models Harm Learning?
This research addresses the problem of optimizing LLM use in education to enhance learning outcomes, particularly for students and educators, by identifying specific usage patterns that can help or harm, though it is incremental as it builds on prior debates.
The study investigated the impact of large language models (LLMs) on learning outcomes, finding no overall effect but that usage behavior matters: substituting activities with LLMs increases topic volume but decreases understanding, while complementing activities increases understanding without boosting volume, and LLMs widen gaps between students with low and high prior knowledge.
The effect of large language models (LLMs) in education is debated: Previous research shows that LLMs can help as well as hurt learning. In two pre-registered and incentivized laboratory experiments, we find no effect of LLMs on overall learning outcomes. In exploratory analyses and a field study, we provide evidence that the effect of LLMs on learning outcomes depends on usage behavior. Students who substitute some of their learning activities with LLMs (e.g., by generating solutions to exercises) increase the volume of topics they can learn about but decrease their understanding of each topic. Students who complement their learning activities with LLMs (e.g., by asking for explanations) do not increase topic volume but do increase their understanding. We also observe that LLMs widen the gap between students with low and high prior knowledge. While LLMs show great potential to improve learning, their use must be tailored to the educational context and students' needs.