AI-TA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs
This addresses the high human cost of handling student questions in computing courses with growing enrollments, though it is incremental as it builds on existing LLM methods.
The paper tackles the challenge of scaling question-answering for online educational platforms by developing an AI assistant using open-source LLMs, achieving a 30% improvement in answer quality through techniques like RAG, SFT, and DPO on a dataset from an introductory CS course.
Responding to the thousands of student questions on online QA platforms each semester has a considerable human cost, particularly in computing courses with rapidly growing enrollments. To address the challenges of scalable and intelligent question-answering (QA), we introduce an innovative solution that leverages open-source Large Language Models (LLMs) from the LLaMA-2 family to ensure data privacy. Our approach combines augmentation techniques such as retrieval augmented generation (RAG), supervised fine-tuning (SFT), and learning from human preferences data using Direct Preference Optimization (DPO). Through extensive experimentation on a Piazza dataset from an introductory CS course, comprising 10,000 QA pairs and 1,500 pairs of preference data, we demonstrate a significant 30% improvement in the quality of answers, with RAG being a particularly impactful addition. Our contributions include the development of a novel architecture for educational QA, extensive evaluations of LLM performance utilizing both human assessments and LLM-based metrics, and insights into the challenges and future directions of educational data processing. This work paves the way for the development of AI-TA, an intelligent QA assistant customizable for courses with an online QA platform