Active Testing: An Unbiased Evaluation Method for Distantly Supervised Relation Extraction
This addresses evaluation fairness and optimization guidance for researchers in relation extraction, though it is incremental as it focuses on improving evaluation rather than the extraction method itself.
The paper tackles the problem of biased performance evaluation in distantly supervised relation extraction due to low-quality test sets, proposing an active testing method that uses noisy test data and manual annotations to achieve approximately unbiased evaluations.
Distant supervision has been a widely used method for neural relation extraction for its convenience of automatically labeling datasets. However, existing works on distantly supervised relation extraction suffer from the low quality of test set, which leads to considerable biased performance evaluation. These biases not only result in unfair evaluations but also mislead the optimization of neural relation extraction. To mitigate this problem, we propose a novel evaluation method named active testing through utilizing both the noisy test set and a few manual annotations. Experiments on a widely used benchmark show that our proposed approach can yield approximately unbiased evaluations for distantly supervised relation extractors.