DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extraction
This provides a more challenging benchmark for researchers in multilingual NLP, enabling better utilization of diverse data across languages, though it is incremental as it builds on existing DS-RE methods.
The authors tackled the lack of realistic multilingual datasets for distantly supervised relation extraction by creating DiS-ReX, a new dataset with over 1.5 million sentences across 4 languages and 37 relation classes, which addresses issues like lack of no-relation sentences and overestimation of model performance.
Distant supervision (DS) is a well established technique for creating large-scale datasets for relation extraction (RE) without using human annotations. However, research in DS-RE has been mostly limited to the English language. Constraining RE to a single language inhibits utilization of large amounts of data in other languages which could allow extraction of more diverse facts. Very recently, a dataset for multilingual DS-RE has been released. However, our analysis reveals that the proposed dataset exhibits unrealistic characteristics such as 1) lack of sentences that do not express any relation, and 2) all sentences for a given entity pair expressing exactly one relation. We show that these characteristics lead to a gross overestimation of the model performance. In response, we propose a new dataset, DiS-ReX, which alleviates these issues. Our dataset has more than 1.5 million sentences, spanning across 4 languages with 36 relation classes + 1 no relation (NA) class. We also modify the widely used bag attention models by encoding sentences using mBERT and provide the first benchmark results on multilingual DS-RE. Unlike the competing dataset, we show that our dataset is challenging and leaves enough room for future research to take place in this field.