REflex: Flexible Framework for Relation Extraction in Multiple Domains
This work addresses reproducibility and comparison issues in relation extraction research, providing a flexible framework and recommendations, but it is incremental as it builds on existing methods without introducing new paradigms.
The authors tackled the problem of inconsistent and non-reproducible relation extraction experiments by building a unifying framework, REflex, and systematically exploring modeling, pre-processing, and training methods across three domains, finding that pre-processing choices significantly impact performance and that lack of such details hinders fair comparisons.
Systematic comparison of methods for relation extraction (RE) is difficult because many experiments in the field are not described precisely enough to be completely reproducible and many papers fail to report ablation studies that would highlight the relative contributions of their various combined techniques. In this work, we build a unifying framework for RE, applying this on three highly used datasets (from the general, biomedical and clinical domains) with the ability to be extendable to new datasets. By performing a systematic exploration of modeling, pre-processing and training methodologies, we find that choices of pre-processing are a large contributor performance and that omission of such information can further hinder fair comparison. Other insights from our exploration allow us to provide recommendations for future research in this area.