$\textit{BenchIE}^{FL}$ : A Manually Re-Annotated Fact-Based Open Information Extraction Benchmark
This work addresses the need for objective benchmarks in the OIE field, but it is incremental as it builds upon and refines an existing benchmark.
The authors tackled the problem of evaluating Open Information Extraction (OIE) systems by identifying issues in the existing BenchIE benchmark, and they proposed BenchIE^FL, a manually re-annotated benchmark that reduces errors and omissions to enable more accurate performance assessment of OIE extractors.
Open Information Extraction (OIE) is a field of natural language processing that aims to present textual information in a format that allows it to be organized, analyzed and reflected upon. Numerous OIE systems are developed, claiming ever-increasing performance, marking the need for objective benchmarks. BenchIE is the latest reference we know of. Despite being very well thought out, we noticed a number of issues we believe are limiting. Therefore, we propose $\textit{BenchIE}^{FL}$, a new OIE benchmark which fully enforces the principles of BenchIE while containing fewer errors, omissions and shortcomings when candidate facts are matched towards reference ones. $\textit{BenchIE}^{FL}$ allows insightful conclusions to be drawn on the actual performance of OIE extractors.