Knowledge-Augmented Language Models for Cause-Effect Relation Classification
This work addresses cause-effect classification for NLP applications, but it is incremental as it builds on existing knowledge augmentation methods.
The authors tackled cause-effect relation classification by augmenting pretrained language models with commonsense knowledge, resulting in models that outperformed baselines on benchmarks like COPA, BCOPA-CE, and TCR without architectural changes.
Previous studies have shown the efficacy of knowledge augmentation methods in pretrained language models. However, these methods behave differently across domains and downstream tasks. In this work, we investigate the augmentation of pretrained language models with commonsense knowledge in the cause-effect relation classification and commonsense causal reasoning tasks. After automatically verbalizing ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, and GLUCOSE, a dataset of implicit commonsense causal knowledge, we continually pretrain BERT and RoBERTa with the verbalized data. Then we evaluate the resulting models on cause-effect pair classification and answering commonsense causal reasoning questions. Our results show that continually pretrained language models augmented with commonsense knowledge outperform our baselines on two commonsense causal reasoning benchmarks, COPA and BCOPA-CE, and the Temporal and Causal Reasoning (TCR) dataset, without additional improvement in model architecture or using quality-enhanced data for fine-tuning.