Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions
This addresses the need for interpretability in NLP by providing a causal analysis method for researchers and practitioners to better understand model behavior, though it is incremental in applying causality to existing data statistics.
The authors tackled the problem of understanding how training data influences language model predictions by developing a causal framework that estimates effects of data statistics like co-occurrence counts without retraining models, showing these statistics significantly affect predictions, indicating reliance on shallow heuristics.
Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. But what exactly in the training data causes a model to make a certain prediction? We seek to answer this question by providing a language for describing how training data influences predictions, through a causal framework. Importantly, our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone. Addressing the problem of extracting factual knowledge from pretrained language models (PLMs), we focus on simple data statistics such as co-occurrence counts and show that these statistics do influence the predictions of PLMs, suggesting that such models rely on shallow heuristics. Our causal framework and our results demonstrate the importance of studying datasets and the benefits of causality for understanding NLP models.