Compositional Evaluation on Japanese Textual Entailment and Similarity
This addresses a gap in multilingual evaluation for Japanese, which is typologically different from English, but is incremental as it builds on existing English datasets.
The researchers tackled the lack of Japanese datasets for Natural Language Inference (NLI) and Semantic Textual Similarity (STS) by introducing JSICK, a manually translated dataset from English, and a stress-test dataset to evaluate language models' sensitivity to word order and case particles. The results showed that current pre-trained language models are insensitive to these linguistic features.
Natural Language Inference (NLI) and Semantic Textual Similarity (STS) are widely used benchmark tasks for compositional evaluation of pre-trained language models. Despite growing interest in linguistic universals, most NLI/STS studies have focused almost exclusively on English. In particular, there are no available multilingual NLI/STS datasets in Japanese, which is typologically different from English and can shed light on the currently controversial behavior of language models in matters such as sensitivity to word order and case particles. Against this background, we introduce JSICK, a Japanese NLI/STS dataset that was manually translated from the English dataset SICK. We also present a stress-test dataset for compositional inference, created by transforming syntactic structures of sentences in JSICK to investigate whether language models are sensitive to word order and case particles. We conduct baseline experiments on different pre-trained language models and compare the performance of multilingual models when applied to Japanese and other languages. The results of the stress-test experiments suggest that the current pre-trained language models are insensitive to word order and case marking.