JETHICS: Japanese Ethics Understanding Evaluation Dataset
This provides a domain-specific evaluation tool for AI ethics in Japanese, but it is incremental as it adapts an existing English dataset.
The authors tackled the problem of evaluating ethics understanding in AI models by creating JETHICS, a Japanese dataset with 78K examples based on normative theories and commonsense morality, and found that even GPT-4o scored only about 0.7 on average, with the best Japanese LLM at around 0.5, showing significant room for improvement.
In this work, we propose JETHICS, a Japanese dataset for evaluating ethics understanding of AI models. JETHICS contains 78K examples and is built by following the construction methods of the existing English ETHICS dataset. It includes four categories based normative theories and concepts from ethics and political philosophy; and one representing commonsense morality. Our evaluation experiments on non-proprietary large language models (LLMs) and on GPT-4o reveal that even GPT-4o achieves only an average score of about 0.7, while the best-performing Japanese LLM attains around 0.5, indicating a relatively large room for improvement in current LLMs.