IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models
This work addresses the need for better benchmarks to assess relational knowledge in pre-trained language models for the NLP community, but it is incremental as it builds on existing OIE methods and datasets.
The authors tackled the problem of evaluating open relational knowledge in pre-trained language models by creating a new open information extraction (OIE) benchmark and turning these models into zero-shot OIE systems, achieving competitive performance, such as outperforming state-of-the-art supervised methods on factual OIE datasets without training.
We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM). Recent studies have demonstrated that pre-trained LMs, such as BERT and GPT, may store linguistic and relational knowledge. In particular, LMs are able to answer ``fill-in-the-blank'' questions when given a pre-defined relation category. Instead of focusing on pre-defined relations, we create an OIE benchmark aiming to fully examine the open relational information present in the pre-trained LMs. We accomplish this by turning pre-trained LMs into zero-shot OIE systems. Surprisingly, pre-trained LMs are able to obtain competitive performance on both standard OIE datasets (CaRB and Re-OIE2016) and two new large-scale factual OIE datasets (TAC KBP-OIE and Wikidata-OIE) that we establish via distant supervision. For instance, the zero-shot pre-trained LMs outperform the F1 score of the state-of-the-art supervised OIE methods on our factual OIE datasets without needing to use any training sets. Our code and datasets are available at https://github.com/cgraywang/IELM