Few Clean Instances Help Denoising Distant Supervision
This work addresses noise issues in relation extraction for NLP researchers, offering an incremental improvement with a model-agnostic approach.
The paper tackles the problem of noisy data in distantly supervised relation extraction by showing that a small clean dataset can improve model evaluation and robustness, achieving strong performances on real and synthetic datasets.
Existing distantly supervised relation extractors usually rely on noisy data for both model training and evaluation, which may lead to garbage-in-garbage-out systems. To alleviate the problem, we study whether a small clean dataset could help improve the quality of distantly supervised models. We show that besides getting a more convincing evaluation of models, a small clean dataset also helps us to build more robust denoising models. Specifically, we propose a new criterion for clean instance selection based on influence functions. It collects sample-level evidence for recognizing good instances (which is more informative than loss-level evidence). We also propose a teacher-student mechanism for controlling purity of intermediate results when bootstrapping the clean set. The whole approach is model-agnostic and demonstrates strong performances on both denoising real (NYT) and synthetic noisy datasets.