Test-Time Reasoners Are Strategic Multiple-Choice Test-Takers
This challenges the assumption that partial-input success in MCQA is always a flaw, offering insights for improving evaluation methods in AI.
The study investigated whether large language models (LLMs) use shallow strategies like ignoring questions in multiple-choice question answering (MCQA) by analyzing reasoning traces, finding that reasoning often boosts accuracy in both full and choices-only inputs, with choices-only success not strongly linked to reasoning length and sometimes involving less problematic strategies like inferring missing questions.
Large language models (LLMs) now give reasoning before answering, excelling in tasks like multiple-choice question answering (MCQA). Yet, a concern is that LLMs do not solve MCQs as intended, as work finds LLMs sans reasoning succeed in MCQA without using the question, i.e., choices-only. Such partial-input success is often deemed problematic, but reasoning traces could reveal if these strategies are truly shallow in choices-only settings. To study these strategies, reasoning LLMs solve MCQs in full and choices-only inputs; test-time reasoning often boosts accuracy on full and in choices-only half the time. While possibly due to shallow shortcuts, choices-only success is barely affected by the length of reasoning traces, and after finding traces pass faithfulness tests, we show they use less problematic strategies like inferring missing questions. In all, we challenge claims that partial-input success is always a flaw, so we discuss how reasoning traces could separate problematic data from less problematic reasoning.