A Comparative Study of Student Perspectives on Technical Writing Feedback Quality: Evaluating LLMs, SLMs, and Humans in Computer Science Topics
For CS educators, this work demonstrates that local SLMs can provide privacy-preserving, cost-effective feedback for foundational tasks, enabling a tiered pedagogical approach.
This study compares student perceptions of feedback quality from a local SLM (quantized Llama-3.1), GPT-4, and human instructors in CS courses (N=263). The local SLM matched commercial LLMs and was rated higher for readability and actionability in technical courses, while human feedback was preferred for specialized writing tasks.
To address the scalability of feedback in computer science while mitigating the privacy and cost limitations of commercial Large Language Models (LLMs), this study evaluates a locally hosted Small Language Model (SLM). We deployed a quantized Llama-3.1, GPT-4, and human instructors across introductory programming (N=176), operating systems (N=80), and a writing seminar (N=7). Mixed-methods analysis of student perceptions reveals that while the local SLM matched commercial LLMs and was rated higher by students for readability and actionability in technical courses, human feedback remained more favoured for highly specialized writing tasks. We demonstrate that local SLMs offer a privacy-preserving, zero-marginal-cost alternative for foundational feedback, supporting a tiered pedagogical framework where AI handles structural guidance while instructors focus on high-level conceptual scaffolding.