Counterfactual Multihop QA: A Cause-Effect Approach for Reducing Disconnected Reasoning
This addresses a specific issue in multi-hop QA for NLP researchers, offering an incremental improvement by reducing reliance on shortcuts.
The paper tackles the problem of disconnected reasoning in multi-hop QA, where models rely on shortcuts instead of true multi-hop reasoning, and proposes a counterfactual causal-effect approach that achieves a 5.8% improvement in Supp_s score on HotpotQA.
Multi-hop QA requires reasoning over multiple supporting facts to answer the question. However, the existing QA models always rely on shortcuts, e.g., providing the true answer by only one fact, rather than multi-hop reasoning, which is referred as $\textit{disconnected reasoning}$ problem. To alleviate this issue, we propose a novel counterfactual multihop QA, a causal-effect approach that enables to reduce the disconnected reasoning. It builds upon explicitly modeling of causality: 1) the direct causal effects of disconnected reasoning and 2) the causal effect of true multi-hop reasoning from the total causal effect. With the causal graph, a counterfactual inference is proposed to disentangle the disconnected reasoning from the total causal effect, which provides us a new perspective and technology to learn a QA model that exploits the true multi-hop reasoning instead of shortcuts. Extensive experiments have conducted on the benchmark HotpotQA dataset, which demonstrate that the proposed method can achieve notable improvement on reducing disconnected reasoning. For example, our method achieves 5.8% higher points of its Supp$_s$ score on HotpotQA through true multihop reasoning. The code is available at supplementary material.