Differentiating Choices via Commonality for Multiple-Choice Question Answering
This addresses the problem of improving accuracy in MCQA for AI systems, particularly in scenarios with subtle differences between choices, though it is incremental as it builds on existing attention mechanisms.
The paper tackles the challenge of multiple-choice question answering when choices are semantically similar by proposing DCQA, a model that differentiates choices by identifying and eliminating commonalities, resulting in consistent outperformance of baseline models across five benchmarks.
Multiple-choice question answering (MCQA) becomes particularly challenging when all choices are relevant to the question and are semantically similar. Yet this setting of MCQA can potentially provide valuable clues for choosing the right answer. Existing models often rank each choice separately, overlooking the context provided by other choices. Specifically, they fail to leverage the semantic commonalities and nuances among the choices for reasoning. In this paper, we propose a novel MCQA model by differentiating choices through identifying and eliminating their commonality, called DCQA. Our model captures token-level attention of each choice to the question, and separates tokens of the question attended to by all the choices (i.e., commonalities) from those by individual choices (i.e., nuances). Using the nuances as refined contexts for the choices, our model can effectively differentiate choices with subtle differences and provide justifications for choosing the correct answer. We conduct comprehensive experiments across five commonly used MCQA benchmarks, demonstrating that DCQA consistently outperforms baseline models. Furthermore, our case study illustrates the effectiveness of the approach in directing the attention of the model to more differentiating features.