Investigating Counterclaims in Causality Extraction from Text
This addresses a gap in causality extraction research for NLP applications, but it is incremental as it builds on existing datasets and methods.
The paper tackles the neglect of counterclaims in causality extraction from text by developing a new dataset that integrates concausal statements, achieving a Cohen's κ of 0.74 for inter-annotator agreement and showing that models trained on this dataset can effectively distinguish between pro- and concausality, reducing misclassification errors.
Research on causality extraction from text has so far almost entirely neglected counterclaims. Existing causality extraction datasets focus solely on "procausal" claims, i.e., statements that support a relationship. "Concausal" claims, i.e., statements that refute a relationship, are entirely ignored or even accidentally annotated as procausal. We address this shortcoming by developing a new dataset that integrates concausality. Based on an extensive literature review, we first show that concausality is an integral part of causal reasoning on incomplete knowledge. We operationalize this theory in the form of a rigorous guideline for annotation and then augment the Causal News Corpus with concausal statements, obtaining a substantial inter-annotator agreement of Cohen's $κ=0.74$. To demonstrate the importance of integrating concausal statements, we show that models trained without concausal relationships tend to misclassify these as procausal instead. Based on our new dataset, this mistake can be mitigated, enabling transformers to effectively distinguish pro- and concausality.