POE: Process of Elimination for Multiple Choice Reasoning
This addresses the challenge of improving reasoning accuracy for language models in multiple choice tasks, though it is incremental as it builds on existing in-context learning methods.
The paper tackles the problem of language models performing poorly on multiple choice reasoning tasks by proposing a two-step scoring method called Process of Elimination (POE), which first eliminates wrong options and then makes a final prediction, showing effectiveness in zero-shot experiments on 8 reasoning tasks with particular gains in logical reasoning.
Language models (LMs) are capable of conducting in-context learning for multiple choice reasoning tasks, but the options in these tasks are treated equally. As humans often first eliminate wrong options before picking the final correct answer, we argue a similar two-step strategy can make LMs better at these tasks. To this end, we present the Process of Elimination (POE), a two-step scoring method. In the first step, POE scores each option, and eliminates seemingly wrong options. In the second step, POE masks these wrong options, and makes the final prediction from the remaining options. Zero-shot experiments on 8 reasoning tasks illustrate the effectiveness of POE, and a following analysis finds our method to be especially performant on logical reasoning tasks. We further analyze the effect of masks, and show that POE applies to few-shot settings and large language models (LLMs) like ChatGPT.