Can Large Language Models Robustly Perform Natural Language Inference for Japanese Comparatives?
This work addresses the challenge of LLM robustness in NLI for under-resourced languages like Japanese, but it is incremental as it builds on existing NLI evaluation methods.
The study tackled the problem of evaluating the robustness of Large Language Models (LLMs) in performing Natural Language Inference (NLI) for Japanese comparatives, revealing that model performance is sensitive to prompt formats and struggles with unique Japanese linguistic phenomena, with prompts containing logical semantic representations improving accuracy on challenging inference problems.
Large Language Models (LLMs) perform remarkably well in Natural Language Inference (NLI). However, NLI involving numerical and logical expressions remains challenging. Comparatives are a key linguistic phenomenon related to such inference, but the robustness of LLMs in handling them, especially in languages that are not dominant in the models' training data, such as Japanese, has not been sufficiently explored. To address this gap, we construct a Japanese NLI dataset that focuses on comparatives and evaluate various LLMs in zero-shot and few-shot settings. Our results show that the performance of the models is sensitive to the prompt formats in the zero-shot setting and influenced by the gold labels in the few-shot examples. The LLMs also struggle to handle linguistic phenomena unique to Japanese. Furthermore, we observe that prompts containing logical semantic representations help the models predict the correct labels for inference problems that they struggle to solve even with few-shot examples.