Aristotelian Virtue Profiling of LLMs through Ethical Dilemmas
For AI ethics researchers and developers, this provides a structured, validated method to measure and compare the ethical priorities of LLMs, though it is incremental as it applies existing virtue ethics to LLM evaluation.
The authors introduce VirtueMap, a framework for profiling LLMs according to Aristotelian virtues (Practical Wisdom, Justice, Truthfulness, Courage, Temperance) by having them rank responses to ethical dilemmas. They find high rank consistency (90.3%) across nine LLM families, with largest differences on Courage, Temperance, and Justice.
Large Language Models (LLMs) often face ethical tradeoffs in which several responses may be defensible but express different priorities, such as fairness, honesty, courage, or restraint. We introduce VirtueMap, a framework for describing these patterns through an Aristotelian virtue-ethics lens. Instead of asking for a single correct answer, VirtueMap asks humans or LLMs to rank all five responses to each of seven general, non-lethal, non-political, and non-religious ethical dilemmas. To define the reference orderings used for scoring, we first proposed, for each dilemma and virtue, an ordering of the five responses from most to least expressive of that virtue. We then collected more than 100 respondent evaluations per ordering and retained it as operational ground truth only when at least 95% confirmed it. Rankings are scored against these retained orderings using normalized Borda alignment, yielding profiles over Practical Wisdom, Justice, Truthfulness, Courage, and Temperance. We apply VirtueMap to nine LLM families in a repeated-run evaluation and find high mean rank consistency (90.3%), with the largest differences appearing on Courage, Temperance, and Justice. We also release an interactive website that computes profiles locally in the browser and compares respondents with measured LLM profiles.