D-GEN: Automatic Distractor Generation and Evaluation for Reliable Assessment of Generative Model
This work addresses the problem of labor-intensive distractor generation for researchers and practitioners in AI evaluation, offering an incremental improvement in automating and standardizing multiple-choice assessments.
The paper tackled the challenge of evaluating generative models by introducing D-GEN, an open-source model that automatically generates high-quality distractors for multiple-choice formats, achieving high ranking consistency (Spearman's rho 0.99, Kendall's tau 0.94) and matching entropy distributions of ground-truth distractors.
Evaluating generative models with open-ended generation is challenging due to inconsistencies in response formats. Multiple-choice (MC) evaluation mitigates this issue, but generating high-quality distractors is time-consuming and labor-intensive. We introduce D-GEN, the first open-source distractor generator model that transforms open-ended data into an MC format. To evaluate distractor quality, we propose two novel methods: (1) ranking alignment, ensuring generated distractors retain the discriminatory power of ground-truth distractors, and (2) entropy analysis, comparing model confidence distributions. Our results show that D-GEN preserves ranking consistency (Spearman's rho 0.99, Kendall's tau 0.94) and closely matches the entropy distribution of ground-truth distractors. Human evaluation further confirms the fluency, coherence, distractiveness, and incorrectness. Our work advances robust and efficient distractor generation with automated evaluation, setting a new standard for MC evaluation.