Exploiting News Article Structure for Automatic Corpus Generation of Entailment Datasets
This addresses the problem of evaluating NLP models in low-resource languages like Filipino, though it is incremental as it applies known techniques to a new domain.
The paper tackles the lack of benchmark datasets for low-resource languages in NLP by automatically generating a Natural Language Inference dataset for Filipino using news articles, resulting in the creation of NewsPH-NLI and new pretrained transformers that are benchmarked against existing methods.
Transformers represent the state-of-the-art in Natural Language Processing (NLP) in recent years, proving effective even in tasks done in low-resource languages. While pretrained transformers for these languages can be made, it is challenging to measure their true performance and capacity due to the lack of hard benchmark datasets, as well as the difficulty and cost of producing them. In this paper, we present three contributions: First, we propose a methodology for automatically producing Natural Language Inference (NLI) benchmark datasets for low-resource languages using published news articles. Through this, we create and release NewsPH-NLI, the first sentence entailment benchmark dataset in the low-resource Filipino language. Second, we produce new pretrained transformers based on the ELECTRA technique to further alleviate the resource scarcity in Filipino, benchmarking them on our dataset against other commonly-used transfer learning techniques. Lastly, we perform analyses on transfer learning techniques to shed light on their true performance when operating in low-data domains through the use of degradation tests.