Towards Realistic Few-Shot Relation Extraction: A New Meta Dataset and Evaluation
This work addresses the challenge of few-shot relation extraction for NLP researchers by providing a realistic benchmark, but it is incremental as it builds on existing datasets without proposing a new method.
The authors tackled the problem of few-shot relation extraction by introducing a new meta dataset derived from existing datasets under realistic assumptions, and found that no existing method performed well, with overall low performance indicating a need for future research.
We introduce a meta dataset for few-shot relation extraction, which includes two datasets derived from existing supervised relation extraction datasets NYT29 (Takanobu et al., 2019; Nayak and Ng, 2020) and WIKIDATA (Sorokin and Gurevych, 2017) as well as a few-shot form of the TACRED dataset (Sabo et al., 2021). Importantly, all these few-shot datasets were generated under realistic assumptions such as: the test relations are different from any relations a model might have seen before, limited training data, and a preponderance of candidate relation mentions that do not correspond to any of the relations of interest. Using this large resource, we conduct a comprehensive evaluation of six recent few-shot relation extraction methods, and observe that no method comes out as a clear winner. Further, the overall performance on this task is low, indicating substantial need for future research. We release all versions of the data, i.e., both supervised and few-shot, for future research.