Inducing Human-like Biases in Moral Reasoning Language Models
This work addresses the problem of aligning AI with human moral reasoning for AI safety and cognitive science, but it is incremental as it builds on existing benchmarks and methods without breakthrough results.
The study investigated whether fine-tuning large language models (LLMs) on human moral reasoning data (behavioral and fMRI) could induce human-like biases, but found that BrainScores did not significantly improve despite larger models performing better on accuracy metrics.
In this work, we study the alignment (BrainScore) of large language models (LLMs) fine-tuned for moral reasoning on behavioral data and/or brain data of humans performing the same task. We also explore if fine-tuning several LLMs on the fMRI data of humans performing moral reasoning can improve the BrainScore. We fine-tune several LLMs (BERT, RoBERTa, DeBERTa) on moral reasoning behavioral data from the ETHICS benchmark [Hendrycks et al., 2020], on the moral reasoning fMRI data from Koster-Hale et al. [2013], or on both. We study both the accuracy on the ETHICS benchmark and the BrainScores between model activations and fMRI data. While larger models generally performed better on both metrics, BrainScores did not significantly improve after fine-tuning.