Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions
This addresses a specific issue in LLM evaluation and alignment for researchers and practitioners, offering a practical method to improve performance on multiple-choice and free-form questions.
The paper tackles the problem of LLMs failing on multiple-choice questions despite having correct knowledge, by identifying a misalignment between knowledge and prediction subspaces in hidden states and introducing a parameter-free intervention called KAPPA to align them. Experiments on Big-Bench-Hard and ARC-Challenge show that KAPPA substantially improves accuracy, outperforming baselines.
Large Language Models (LLMs) often fail on multiple-choice questions (MCQs) despite demonstrating correct knowledge in other contexts, such as free-form generation. To investigate the mechanism underlying this knowledge-prediction gap on MCQs and alleviate it, we conduct a probing analysis and find that residual streams in certain layers contain a subspace spanned by two important bases: a \emph{knowledge basis} that encodes the probability of the ground-truth answer for a given MCQ and a \emph{prediction basis} that encodes the probability of the answer choice predicted by the model. We observe that incorrect predictions arise from a misalignment of the model's hidden states along these two bases. Hence, we introduce \textbf{KAPPA} (Knowledge-Aligned Prediction through Projection-based Adjustment), a parameter-free intervention that transforms the hidden states to align the prediction coordinate with the knowledge coordinate within this subspace. Experiments on binary-choice reformulations of Big-Bench-Hard and ARC-Challenge show that KAPPA substantially improves accuracy and consistently outperforms baselines. While optimal subspaces differ across tasks, subspaces generalize to some extent, as supported by cross-dataset experiments. Moreover, KAPPA extends its effectiveness to free-form questions beyond MCQs. Our work provides a new geometric understanding of the knowledge-prediction gap and offers a practical method for better aligning model behavior with its latent knowledge.