He Thinks He Knows Better than the Doctors: BERT for Event Factuality Fails on Pragmatics
This work highlights a critical limitation in NLP models for factuality prediction, which is important for applications like misinformation detection, but it is incremental as it builds on existing BERT evaluations.
The study evaluated BERT's performance on event factuality prediction across English datasets, finding that while it achieves strong results by exploiting surface patterns, it fails on instances requiring pragmatic reasoning, indicating a lack of robustness in such systems.
We investigate how well BERT performs on predicting factuality in several existing English datasets, encompassing various linguistic constructions. Although BERT obtains a strong performance on most datasets, it does so by exploiting common surface patterns that correlate with certain factuality labels, and it fails on instances where pragmatic reasoning is necessary. Contrary to what the high performance suggests, we are still far from having a robust system for factuality prediction.