Extended Japanese Commonsense Morality Dataset with Masked Token and Label Enhancement
This work addresses the problem of cultural bias in AI morality datasets for Japanese contexts, though it is incremental as it builds on an existing dataset.
The authors tackled the lack of cultural diversity in AI moral reasoning by expanding the Japanese Commonsense Morality dataset from 13,975 to 31,184 sentences using a novel Masked Token and Label Enhancement method, resulting in a model that achieved an F1 score of 0.857, outperforming previous methods and showing significant improvement in complex Japanese cultural tasks from 0.681 to 0.756.
Rapid advancements in artificial intelligence (AI) have made it crucial to integrate moral reasoning into AI systems. However, existing models and datasets often overlook regional and cultural differences. To address this shortcoming, we have expanded the JCommonsenseMorality (JCM) dataset, the only publicly available dataset focused on Japanese morality. The Extended JCM (eJCM) has grown from the original 13,975 sentences to 31,184 sentences using our proposed sentence expansion method called Masked Token and Label Enhancement (MTLE). MTLE selectively masks important parts of sentences related to moral judgment and replaces them with alternative expressions generated by a large language model (LLM), while re-assigning appropriate labels. The model trained using our eJCM achieved an F1 score of 0.857, higher than the scores for the original JCM (0.837), ChatGPT one-shot classification (0.841), and data augmented using AugGPT, a state-of-the-art augmentation method (0.850). Specifically, in complex moral reasoning tasks unique to Japanese culture, the model trained with eJCM showed a significant improvement in performance (increasing from 0.681 to 0.756) and achieved a performance close to that of GPT-4 Turbo (0.787). These results demonstrate the validity of the eJCM dataset and the importance of developing models and datasets that consider the cultural context.