IndoNLI: A Natural Language Inference Dataset for Indonesian
This dataset addresses the lack of resources for Indonesian NLP research, though it is incremental as it adapts existing protocols to a new language.
The authors introduced IndoNLI, the first human-elicited natural language inference dataset for Indonesian, comprising nearly 18K sentence pairs, and found that XLM-R performed best but still had a 13.4% accuracy gap compared to human performance on the expert-annotated test set.
We present IndoNLI, the first human-elicited NLI dataset for Indonesian. We adapt the data collection protocol for MNLI and collect nearly 18K sentence pairs annotated by crowd workers and experts. The expert-annotated data is used exclusively as a test set. It is designed to provide a challenging test-bed for Indonesian NLI by explicitly incorporating various linguistic phenomena such as numerical reasoning, structural changes, idioms, or temporal and spatial reasoning. Experiment results show that XLM-R outperforms other pre-trained models in our data. The best performance on the expert-annotated data is still far below human performance (13.4% accuracy gap), suggesting that this test set is especially challenging. Furthermore, our analysis shows that our expert-annotated data is more diverse and contains fewer annotation artifacts than the crowd-annotated data. We hope this dataset can help accelerate progress in Indonesian NLP research.