STRIVE: A Think & Improve Approach with Iterative Refinement for Enhancing Question Quality Estimation
This addresses the problem of time-consuming and inconsistent question assessment for educators, though it appears incremental as it builds on existing LLM capabilities.
The authors tackled automatic question quality estimation for educators by proposing STRIVE, a method using iterative refinement with multiple LLMs, which improved correlation with human judgments and metrics like relevance and appropriateness compared to baseline methods.
Automatically assessing question quality is crucial for educators as it saves time, ensures consistency, and provides immediate feedback for refining teaching materials. We propose a novel methodology called STRIVE (Structured Thinking and Refinement with multiLLMs for Improving Verified Question Estimation) using a series of Large Language Models (LLMs) for automatic question evaluation. This approach aims to improve the accuracy and depth of question quality assessment, ultimately supporting diverse learners and enhancing educational practices. The method estimates question quality in an automated manner by generating multiple evaluations based on the strengths and weaknesses of the provided question and then choosing the best solution generated by the LLM. Then the process is improved by iterative review and response with another LLM until the evaluation metric values converge. This sophisticated method of evaluating question quality improves the estimation of question quality by automating the task of question quality evaluation. Correlation scores show that using this proposed method helps to improve correlation with human judgments compared to the baseline method. Error analysis shows that metrics like relevance and appropriateness improve significantly relative to human judgments by using STRIVE.