LLaMa-SciQ: An Educational Chatbot for Answering Science MCQ
This is an incremental improvement for college students needing assistance with science MCQs, but it addresses a specific bottleneck in LLMs.
The paper tackled the problem of large language models struggling with mathematical reasoning in multiple-choice questions by developing LLaMa-SciQ, an educational chatbot for STEM students, which achieved 74.5% accuracy on GSM8k and 30% on MATH datasets.
Large Language Models (LLMs) often struggle with tasks requiring mathematical reasoning, particularly multiple-choice questions (MCQs). To address this issue, we developed LLaMa-SciQ, an educational chatbot designed to assist college students in solving and understanding MCQs in STEM fields. We begin by fine-tuning and aligning the models to human preferences. After comparing the performance of Mistral-7B and LLaMa-8B, we selected the latter as the base model due to its higher evaluation accuracy. To further enhance accuracy, we implement Retrieval-Augmented Generation (RAG) and apply quantization to compress the model, reducing inference time and increasing accessibility for students. For mathematical reasoning, LLaMa-SciQ achieved 74.5% accuracy on the GSM8k dataset and 30% on the MATH dataset. However, RAG does not improve performance and even reduces it, likely due to retriever issues or the model's unfamiliarity with context. Despite this, the quantized model shows only a 5% loss in performance, demonstrating significant efficiency improvements.