Affordably Fine-tuned LLMs Provide Better Answers to Course-specific MCQs
This work addresses affordability and resource constraints for educators and students in using LLMs for MCQ answering, representing an incremental improvement in domain-specific applications.
The study tackled the problem of making large language models (LLMs) affordable for educational use by evaluating their performance on course-specific multiple-choice questions (MCQs), finding that smaller, fine-tuned models based on textbook material outperformed larger generic models, with accuracy improvements demonstrated across 162 undergraduate-level questions.
In education, the capability of generating human-like text of Large Language Models (LLMs) inspired work on how they can increase the efficiency of learning and teaching. We study the affordability of these models for educators and students by investigating how LLMs answer multiple-choice questions (MCQs) with respect to hardware constraints and refinement techniques. We explore this space by using generic pre-trained LLMs (the 7B, 13B, and 70B variants of LLaMA-2) to answer 162 undergraduate-level MCQs from a course on Programming Languages (PL) -- the MCQ dataset is a contribution of this work, which we make publicly available. Specifically, we dissect how different factors, such as using readily-available material -- (parts of) the course's textbook -- for fine-tuning and quantisation (to decrease resource usage) can change the accuracy of the responses. The main takeaway is that smaller textbook-based fine-tuned models outperform generic larger ones (whose pre-training requires conspicuous resources), making the usage of LLMs for answering MCQs resource- and material-wise affordable.