ANPMI: Assessing the True Comprehension Capabilities of LLMs for Multiple Choice Questions
This addresses a key issue in accurately measuring natural language understanding for AI researchers, though it is incremental as it refines existing evaluation methods.
The authors tackled the problem that multiple-choice benchmarks for language models conflate comprehension with choice biases, proposing a new metric ANPMI that normalizes Pointwise Mutual Information to better assess true understanding, showing it reduces bias effects in evaluations.
Multiple-choice benchmarks, consisting of various prompts and choices, are among the most widely used methods to assess a language model's natural language understanding capability. Given a specific prompt, we typically compute $P(Choice|Prompt)$ to evaluate how likely a language model is to generate the correct choice compared to incorrect ones. However, we observe that performance measured using this approach reflects not only the model's comprehension of the prompt but also its inherent biases for certain choices regardless of the prompt. This issue makes it challenging to accurately measure a model's natural language understanding, as models may select the answer without fully understanding the prompt. To address this limitation, we propose a novel metric called ANPMI, which normalizes Pointwise Mutual Information (PMI) by $-\log P(Choice)$. ANPMI provides a more accurate assessment of the model's natural language understanding by ensuring that it is challenging to answer a question without properly understanding the prompt.